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1. Introduction 



The purpose of the present paper is to develop a sub-Riemannian calculus on smooth hyper- 
surfaces in a class of nilpotent Lie groups which possess a rich geometry. Such groups arise as 
tangent spaces of Gromov-Hausdorff limits of Riemannian manifolds |CFGj . |Bej . |Monj . and 
since they can be traced back to the foundational paper of Caratheodory |Caj on Carnot ter- 



modynamics, they have been christened Carnot groups by Gromov, see ur 
main motivation is, in a broad sense, the regularity theory of hypersurfaces of constant mean 
curvature in such settings, as well as the applications of the relevant calculus to the study of 
the Bernstein problem. These problems have recently received increasing attention from several 
groups of mathematicians and there exists nowadays a wide literature. The following is only a 
partia l list o f referen c es [HNj . |B1, [FS!?1], |F?^, [F^ . [UP|, |CHMY| . p!| . [XFTVj . [Eg . 
lEEU, IHE!, inSHl!, |D(INP| . PT7| . im^, For an extensive bibliography we refer the 

reader to the recent monog raphs |DGN4j . (CDPTj (see also the forthcoming book IHSj), and to 
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the papers inSIS], |D(INP| . Carnot groups play a pervasive role in analysis, geometry, and in 
various branches of the applied sciences, ranging from problems in optimal control and robotics, 
crystallography, mathematical finance, and neurophysiology of the brain. This latter aspect, 
in particular, has been recently brought to light in some very interesting works of Petitot and 
Tondut [Pel], lETl, EES], and of Citti and Sarti [CHIl, EHll, see also jCMHI], EMSll. These 
latter works have shown that there exists a close link between the way in which the brain chooses 
to complete the missing visual data in the first layer of the cerebral cortex, Vi, and the minimal 
surfaces in a specific sub-Riemannian space, the so-called roto-translation group, arising in the 
mathematical modeling of the visual cortex Vi. 

To describe the content of this paper we recall that during the past century the study of 
minimal surfaces has been one of the main driving forces in mathematics. Such development 
was prompted by the study of the problems of Plateau and Bernstein which has led, as a by- 
product, to the development of the Geometric Measure Theory, see |Fej . |Matlj . |Mat2j . Minimal 
surfaces also play a central role in the positive mass theorem from relativity due to Schoen and 
Yau |SYj . see also the lecture notes |Schj . Given the substantial progress which has occurred 
during the past decade in the theory of subelliptic equations, and in those closely connected 
aspects of geometric measure theory in sub-Riemannian spaces, it seems natural at this point to 
direct the attention to the understanding of those tools which are necessary for the development 
of a rich theory of minimal surfaces. As we mentioned above, in this paper we solely discuss 
hypersurfaces. Minimal manifolds of higher codimension are also of interest and we hope to 
investigate them in future studies. 

In classical geometry a central notion is that of area of a (smooth) hypersurface. Such notion 
was extended by De Giorgi |DGlj . |DG2j . with the introduction of his variational theory of 
perimeters which allowed to assign an "area" also to sets which are not a priori smooth. In 
a Carnot group G there exists a corresponding variational notion of perimeter adapted to the 
horizontal bundle HG (for a brief introduction to Carnot groups we refer the reader to section 
Given a distribution of smooth left-invariant vector fields X = {Xi, Xm} which is an 
orthonormal basis of the horizontal bundle (and therefore it is bracket-generating for TG), and 
an open set O C G, we let 

m 

j^{n) = {(: = y^QXiec^{n,HG) I icloo = sup Id <i}. 

For a function u G Lj^^(Q,), the ff- variation of u with respect to Q, is defined by 

„ m 

VarH(,u;Q) = sup / u "S^Xid dg . 

Ce:F{n) Jg 

A function u G L}{^) is called of bounded i^-variation in Q. if Varuiu;^) < oo. The space 
BVh{^) of functions with bounded //-variation in fi, endowed with the norm 

IWWBVnin) = \W\\L^n) + VarH{u;n) , 

is a Banach space. Similarly to the classical theory (for the latter, see for instance |Gij and |Zj), 
such space constitutes the appropriate replacement of the horizontal Sobolev W^^ (fi) space in 
the study of the relevant minimal surfaces, see |GNj . Let now E C G he a, measurable set, 
C G be an open set. The //-perimeter of E with respect to il. is defined by the equation 



(1.1) 



PHiE;n) = VarnixE^n) , 
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where xe denotes the indicator function of E, see innSl. When^ possesses sufficient regularity, 
e.g. when S = dE is a hypersurface of class C^, then one finds that 



where we have denoted with N the projection of the (non-unit) Riemannian normal to dE 
onto the subbundle HG. It is intereting to note that, in this situation, a useful alternative 
understanding of the //-perimeter can be obtained by blowing-up the (suitably normalized) 
standard surface measure associated with the Riemannian regularization of the sub- Riemannian 
metric, see Theorem 18.51 below. 

A "minimal surface" in O was defined in |GNj as the boundary of a set of least //-perimeter, 
among all those with the same boundaries outside 57. The existence of such "surfaces" (a priori, 
these are just sets of locally finite //-perimeter), and a measure theoretic solution of the Plateau 
problem, were also established in |GNj following the classical approach of De Giorgi |DGlj . 
inns, IdHP] . The natural question arises of whether such measure theoretic minimal surfaces 
have, at least when they are sufficiently smooth, vanishing "mean curvature". This prompts 
to investigate an appropriate notion of mean curvature adapted to the horizontal bundle HG. 
For level sets such a notion was proposed by one of us back in 1997, see jGl| . but its geometric 
content was not obvious. For the Heisenberg group H^, another notion of mean curvature was 
introduced by Pauls in |Paj , who studied the solvability of the Plateau problem by means of the 
Riemannian regularization of the sub-Riemannian metric. For a surface in a three-dimensional 
CR manifold, yet another notion of mean curvature has been recently proposed in |GHM Yj . 
For instance, if the ambient manifold is the Heisenberg group H^, then the mean curvature of a 
surface 5 C is defined as the standard curvature of the curve of intersection of S with the 
horizontal plane passing through the base point. We note that, for surfaces in a Carnot group, 
this same notion of curvature was also already explicitly introduced in |DGJNlj . In this paper, 
given a hypersurface 5 in a Carnot group G, we introduce a second fundamental form on S 
adapted to the horizontal subbundle HG, and a geometric notion of mean curvature of S, and 
we show that the latter coincides with either one of those proposed in [HI], [Eil, and |GHMYj . 
see Propositions E21 UnHl and mH 

In a Carnot group G, with grading of the Lie algebra g = Vi © ... © Vr, we define a smooth 
left-invariant Riemannian metric < •, • > by imposing that the vector fields Xi, X^, ...,Xr^mr^ 
defined in 1)2. 15(1 . be orthonormal, see section [51 We can thus consider the Riemannian connec- 
tion V on G induced by < •,• >. We define the horizontal Levi-Civita connection on G 
by projecting V onto the horizontal bundle HG, see section [SJ We note explicitly that 
is, in essence, Cartan's non-holonomic connection introduced in his address at the Bologna 
International Congress of Mathematicians in 1928, see 

In section ^ given an oriented hypersurface S C G, with Riemannian normal N, we 
define the horizontal normal to S as the projection of N onto the horizontal bundle, and 
the horizontal Gauss map as u = N^/\N^\. Note that \N^\ / at every point which does 
not belong to the characteristic set of S. We recall that the latter is the collection of all 
points g £ S at which HgG C TgS. An important notion is that of horizontal tangent bundle 
HTS to S, whose fiber HTgS at each point g G S\Tjs is defined as the collection of all horizontal 
vectors which are orthogonal to . It can be easily recognized that HTgS = TgS H HgG. To 
obtain a connection on HTS we then project the horizontal Levi-Civita connection on the 
horizontal tangent bundle HTS. More explicitly, for every X,Y £ C^{S; HTS) we define 



(1.2) 




.H 
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where X, Y are any two horizontal vector fields on G such that X = X,Y = Y on S (note that 
the above definition does not depend on the choice of the extensions). Unlike its Riemannian 
counterpart, the connection V-^' Y is not torsion free in general, and therefore it is not Levi- 
Civita in general. This is due to the fact that, given X,Y ^ C^{S; HTS), the projection [X, Y]^ 
onto the horizontal bundle of [X, Y] does not in general belong to the horizontal tangent space 
to S, HTS. We note in passing that an interesting situation in which V-^' Y is Levi-Civita is 
that when G = , the first Heisenberg group, or when G = G;, the four dimensional Engel 
group, see section |21 

Inspired by the Riemannian situation we next project along the horizontal Gauss map 
. In this way we are able to introduce the following notion of horizontal second fundamental 
form on S 

(1.3) 11^'^ {X, Y) = < V^Y, >v^ , 

where X,Y ^ C^{S; HTS). Since [X, y]^ is not in general in HTS, unlike its Riemannian 
predecessor is not symmetric. One has in fact, 

II"'^{X,Y) - II^'^iY,X) = < [X,Y]^,u^ ^ . 

At every point go S^, we define the horizontal mean curvature TC (or //-mean curvature) of 
S as the negative of the trace of the (symmetrized) second fundamental form. If {ei, ...,6^,-1} 
is an othonormal basis of HTS, we thus have 

m—l 

(1.4) n = - Y.<Vle„v^> . 

1=1 

If instead G S5, then we define 7i{go) as the lim T~i{g), whenever such limit exists. A 

hypersurface 5 C G is said to have constant mean-curvature cGMif'H = casa continuous 
function on S. We call S //-minimal if 7/ = on 5. We mention that recently Hladky and Pauls 
|HPj have introduced a notion of mean curvature for hypersurfaces in a class of sub-Riemannian 
spaces which encompasses that of Carnot groups. Their interesting approach can be seen as a 
generalization of the Webster- Tanaka geometric framework for CR manifolds, and systematically 
exploits the Lagrangian framework of Bryant, Griffiths and Grossmann |BGGj . Although the 
notion of second fundamental form in |HPj is different from the one introduced in this paper, we 
notice that, specialized to Carnot groups, their notion of mean curvature coincides with H1.4() . 

Having introduced the notion of //-mean curvature, and //-minimal surface, following the 
steps of the classical developments, it is natural to study questions of regularity, stability, etc. 
It is well-known that in the classical setting when S C M", with the standard surface measure 
da, an essential role in this program is played by the following integration by parts formula, see 

e.g. 

(1.5) f Vf da = {n-l) [ f H u da , 

Js Js 

where V denotes the Levi-Civita connection on 5, / G Cq (5), and H is the mean curvature of S. 
For instance, the fundamental a priori gradient estimates for minimal surfaces are derived from 
(tTT)!) . see |BDMj . In section [TT71 we establish an appropriate generalization of l|1.5|) to the case of 
a hypersurface in a Carnot group. The interesting feature of such intrinsic integration by parts 
formula is that the role of the surface measure is played by the //-perimeter. Furthermore, it 
links the horizontal connection V^'*^ on S to the //-mean curvature of S. The relevant results 
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states that for every f £ Cq{S \ S^), 

(1.6) i"^"'^^ ^ '''' " ' 

where c^'"^ is a vector field on 5 \ with values in the horizontal tangent space HTS, see 
Theorem 110. II This result plays a central role in the establishment of the fundamental first and 
second variation formulas for the i^-perimeter in sections El and 1151 Although 1)1.6(1 formally 
resembles ()1.5() . and in fact it encompasses its Riemannian predecessor, the presence of the vector 
field c^''^ represents a new aspect which reflects the lack of torsion freeness of the connection 
V''^''^, see also Proposition 15. II below. In the Abelian case when G = M", then c^''^ = and we 
recover (|1.5|) . Another interesting situation in which c^''^ = is when 5 is a vertical cylinder 
on the horizontal layer, i.e., when S is locally described by a defining function which depends 
only on the horizontal variables. 

Using the connection V^'*^ we define two differential operators on S, see Definition 111.11 
The former, denoted by Ah,Si is ^ sub-Riemannian version of the classical Laplace-Beltrami 
operator on a manifold. The latter, indicated by Ah,Sj contains an additional drift term, and is 
motivated by the intrinsic integration by parts formula (|1.6|) . Its main raison d'etre, in fact, is 
that a Stokes' type theorem holds for it, see Corollarv lll.3l Formula p.6p implies the following 
identity 

/ <V"^^u,V"'^C>daH = - I u Ah,sC dan . 
Js Js 

for every u E C^{S), and every C £ C^{S\T.s). Using this identity, we introduce a notion of 
sub-harmonicity on S, see Definitions 111.131 111.141 It is an interesting open question to study 
the properties of non-negative sub-harmonic functions on S. For instance, when S is if-minimal, 
do such functions satisfy some kind of sub-mean value formula? 

In Theorem 112.11 we connect the operator A}{,s to the fiow by horizontal mean curvature 
recently introduced by Bonk and Capogna (BC. . We show that, similarly to its Riemannian 
counterpart, such fiow satisfies the following interesting partial differential equation involving 
the horizontal tangential Laplacian Ajj on the hypersurfaces 5* = F(S, t), images of 5 through 
the flow F{-,t), see Theorem 112. II 

dF 

<-^,N> = < Ah,s^F,N> . 

Sections El El and El are entirely devoted to a geometric study of surfaces in the Heisen- 
berg group H^. In this setting, given a surface S with horizontal Gauss map , one easily 
recognizes that HTS is spanned by the single vector fleld (v^)-^. The triple {{i'^)-^ ,T} 
forms an orthonormal moving frame on 5. In section El we establish various geometric identities 
which connect horizontal covariant differentiation along such frame to geometric quantities such 
as the //-mean curvature and its derivatives. 

In section El we use such identities, in combination with some notable integration by parts 
formulas which follow from Theorem llU.il see Lemma 114.81 This lemma plays a crucial role in 
establishing the first and second variation formulas for the //-perimeter measure which constitute 
the main results of the section, see Theorems 114. 31 and 114. 51 The former allows to give a positive 
answer to the question raised above: is a //-minimal surface surface a stationary point of the 
//-perimeter? In Theorem 114.31 we show that for 5 C H^, the first-variation of the //-perimeter 
for a deformation of S along a vector field X £ Cq (5 \ , ) is given by 

(1.7) Vf(5;^) = / n^^^dan, 
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where v = N/\N\ represents the Riemannian Gauss map on 5. In particular, S is stationary if 
and only if it is ff-minimal (see also the less intrinsic first variation formula in Theorem 19. II for 
deformations along the normal N and valid for hypersurfaces in an arbitrary Carnot group). 

The central result of section is Theorem 114.51 which provides a second variation formula 
for the //-perimeter of S. The proof of such formula is considerably more complex than that of 
H1.7|) . and obtaining it has required a substantial effort. Despite such effort we notice, however, 
that Theorem 114. 51 is in practice not as useful as one would hope since it contains several terms 
whose geometric content is not transparent, and which are very difficult to handle. For the 
applications of the second variation formula to the fundamental question of stability it is crucial 
to be able to extract the geometry from Theorem ll4.5l In order to do so one needs to eliminate 
in the integrals involved the various products of covariant derivatives of the projections of the 
testing vector field X along the moving frame {{u^)-^ ,T}. In this endeavor one has to 
choose with extreme care the terms to play one against the other, so to be able to exploit the 
delicate cancelations deriving from the various Lagrangian quantities involved. Section is 
devoted to this goal. In Theorem 115.21 we have succeeded in deriving the following geometric 
second variation formula 

(1.8) VniS;X) = J^llV"""^^!' + i2A-uJ^)F^yaH , 

where 5 C is an //-minimal surface, X is as in (|1.7|) . and we have set 

^ ^ <X,u> 
< u^,u > 

The reader should compare H1.8() with the second variation formula on p. 153 in innnj. The 
coefficient 2A-lJ^ of in (HIHl) is a geometric quantity which involves the projection of N along 
T, and its horizontal covariant derivative along the vector field (u^)-^. With (|1.8)) in hands, one 
can attack the fundamental question of the stability. A non-characteristic //-minimal surface S 
is called stable if V/^(5; X) > for any X G Cq{S, H^). In view of (|1.8|) we see that a surface S 
is stable if and only if the following stability inequality holds on S 

(1.9) [ (aj2 _ 2A)F^daH < [ jV^-^Fp dan ■ 

Js Js 

We emphasize that one can think of H1.9() as a Hardy type inequality on S. One should 

compare (|1.9() with its Riemannian counterpart, see e.g. inequality (1.105) in |CMj for normal 

deformations. 

We conclude this introduction by emphasizing that the study of the stability is an important 
new aspect in the sub-Riemannian Bernstein problem. To clarify this point we recall the well- 
known fact that in the classical Bernstein problem, stability does not apparently play any role. 
This is due to the fact that the area functional for a graph Xn+i = u{x), x € Q C M", 

A{u) = [ Vl + |Vu|2 dx , 

is convex. As a consequence, a critical point of A{u) is also a local minimizer, and there- 
fore stable. By contrast, the sub-Riemannian area functional, the //-perimeter (jl.ip . is not 
convex, see |D(tNP| . and the resulting Euler-Lagrange equation is not elliptic, but degenerate 
hyperbolic (-elliptic). Using the stability inequality (|1.9|) . it has been recently shown in ^DGNS] 
that, contrarily to what was believed by several experts, the entire //-minimal graph x = yt in 
M.^, which has empty characteristic locus, is in fact unstable. This discovery has underscored the 
role of the stability in the sub-Riemannian Bernstein problem and opened the way to the solu- 
tion of the latter. Subsequently, in fact, this result has been generalized in |DGNPj . where it has 
been proved the instability of every graph in of the type x = yG{t), with y E R, t G / C M, 
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with G G C'^{I), and such that G' > on some subinterval J C I. On the other hand, it has 
also been shown in |D(TNPj that every entire i?-minimal graph in H^, with empty characteristic 
locus, and which is not itself a vertical plane ax + by = 7, after possibly a left-translation and 
a rotation about the t-axis, contains a graphical strip of the type x = yG{t), with G' > on 
some subinterval J C M. Combining these two results, the authors have obtained a solution of 
the following sub-Riemannian Bernstein problem: The only stable H-minimal entire graphs in 
, with empty characteristic locus, are the vertical planes. The ideas in |DGJN3j have also been 
used in the recent paper jBSV) to prove a similar Bernstein type theorem for the entire intrinsic 
graphs. 



2. Carnot groups 

In this section we collect some of the basic geometric facts about Carnot groups. We partic- 
ularly emphasize those properties which are useful in this paper. For more extensive sources we 
refer the reader to [Sill, [ES, ESI, Eirj, lETl-[E3j, jWT] . [B^ . [Urol], [nro2j . [Monj, jHS- A 

sub-Riemannian space is a triple (M, HM, d) constituted by a connected Riemannian manifold 
M, with Riemannian distance d-ji, a subbundle of the tangent bundle HM C TM, and the 
Carnot-Caratheodory (CC) distance d generated by HM. Such distance is defined by minimiz- 
ing only on those absolutely continuous paths 7 whose tangent vector j'{t) belongs to H^^^^M, 
see |NSWj . |Bej . Riemannian manifolds are a special example of sub-Riemannian spaces. They 
correspond to the case HM = TM. The tangent space of a sub-Riemannian space is itself 
a sub-Riemannian space (or a quotient of such spaces), but of a special type. It is a graded 
Lie group whose Lie algebra is nilpotent. These groups, which owe their name to the foun- 
dational paper of Charatheodory ICal on Carnot thermodynamics, occupy a central position 
in the study of hypoelliptic partial differential equations, harmonic analysis, sub-Riemannian 
geometry, CR geometric function theory, but also in the applied sciences such as mathematical 
finance, neurophysiology of the brain, mechanical engineering. They are called Carnot groups. 

A Carnot group of step r is a connected, simply connected Lie group G whose Lie algebra g 
admits a stratification Q = Vi® ■ ■ ■ ®Vr which is r-nilpotent, i.e., [Fl, Vj] = Vj+i, j = 1, — 1, 
= {0}, j = l,...,r. We assume henceforth that q is endowed with a scalar product < 
•, • >g with respect to which the V^s are mutually orthogonal. A trivial example of (an Abelian) 
Carnot group is G = M", whose Lie algebra admits the trivial stratification = Vi = M". 
The simplest non-Abelian example of a Carnot group of step r = 2 is the {2n + l)-dimensional 
Heisenberg group H", which is described in section |S1 Given a Carnot group G, by the above 
assumptions on the Lie algebra one immediately sees that any basis of the horizontal layer V\ 
generates the whole g. We will respectively denote by 

(2.1) Lg(5') =55', Rg{d) = g' 9 . 

the operators of left- and right-translation by an element g £ G. 

The exponential mapping exp : q ^ G defines an analytic diffeomorphism onto G. We recall 
the important Baker-Campbell-Hausdorff formula, see, e.g., sec. 2. 15 in |V] . 

(2.2) exp{Oexp{r]) = expU + ij + ^ [^,rj] + - [v,[C,r]]]] + ... 
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where the dots indicate commutators of order four and higher. Each element of the layer Vj is 
assigned the formal degree j. Accordingly, one defines dilations on g by the rule 

AaC = a 6 + - + & , 

provided that ^ = + ... + ^j. € Q, with S^j G Vj. Using the exponential mapping exp : g — > G, 
these anisotropic dilations are then tansferred to the group G as follows 

hig) = exp o Ax oexp"^ g . 

Throughout the paper we will indicate by dg the bi-invariant Haar measure on G obtained by 
lifting via the exponential map exp the Lebesgue measure on g. We let rrij = dim Vj, j = 1, r, 
and denote by = mi + ... + rrir the topological dimension of G. One easily checks that 

r 

{do6x){g) = dg, where Q = ^jm^. 

i=i 

The number Q, called the homogeneous dimension of G, plays an important role in the 
analysis of Carnot groups. In the non-Abelian case r > 1, one clearly has Q > N. 

We denote by d{g,g') the CC distance on G associated with the system X. It is well-known 
that d{g,g') is equivalent to the gauge pseudo-metric p{g,g') on G, i.e., there exists a constant 
C = C{G) > such that 

(2.3) Cp{g,g') < d{g,g') < C'^ p{g,g'), g,g'eG, 

see (03, jCh|, [Eil, |JNSWj . The pseudo-distance p{g,g') is defined as follows, see jF2j. 

Let I • I denote the Euclidean distance to the origin on g. For = + ■ ■ ■ + & Q, (,j & Vj , one 
lets 

(2.4) leig = , \9\g = |exp-i g\„ g G G, 
and defines 

(2.5) p{g,g') = \g'^ g'lc- 

Both d and p are invariant under left-translations 

(2.6) d{L,ig'),Lgig")) = d{g',g") , p{Lgig'), Lg{g")) = p{g',g") . 
and homogeneous of degree one 

(2.7) d{Sxig'),dx{g")) = \d{g\g") , p{Sx{g'),Sxig")) = Xp{g',g") . 
Denoting respectively with 

(2.8) B{g, R) = {g' eG\ d{g',g) < R}, Bp{g, R) = {g' e G \ p{g' , g) < R}, 

the CC ball and the gauge pseudo-ball centered at g with radius R, one easily recognizes that 
there exist uj = u){G) > 0, and a = a{G) > such that 

(2.9) 1^(5,^)1 = ^ \Bp{g,R)\ = a R'^ , geG,R>0. 

Let vTj : g — > Vj denote the projection onto the j-th layer of g. Since the exponential map 
exp : g — > G is a global analytic diffeomorphism, we can define analytic maps : G ^ Vj, 
j = 1, ...,r, by letting = ttj o exp~^. As a rule, we will use letters g,g' ,g",go for points in G, 
whereas we will reserve the letters ^, ^Oj for elements of the Lie algebra g. The notation 
{cj^i, ...,ej^rnj}, j = will indicate a fixed orthonormal basis of the j — th layer Vj. For 
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g £ G, the projection of the exponential coordinates of g onto the layer Vj, j = l,...,r, are 
defined as follows 

(2.10) Xj^sig) = < ^j{g),ej,s >3, s = l,...,mj. 

The vector ^j{g) G Vj, j = 1, will be routinely identified with the point 

{Xj^l{g),...,X,^rn,{g)) G M'™^ . 

Since Carnot groups of step r = 2 often play a special role in analysis and geometry, it will be 
convenient to have a simplified notation for objects in the horizontal layer Vi, and in the first 
vertical layer V2- For simplicity, we set m = mi, k = m2, and let 

(2.11) {ei, ... , em} = {ei,i, ... , ei,mj , {ei,...,ek} = {e2,i, ... , 62,^1} • 
We indicate with 

(2.12) Xi{g) =< ^i{g),ei >Q, i = l,...,m, ts{g) = < ^2(9), >g, s = l,...,k, 

the projections of the exponential coordinates of g onto Vi and V2 respectively. Whenever 
convenient, we will identify g £ G with its exponential coordinates 

(2.13) x{g) =^ {xi{g),...,Xm{g),ti{g),...,tk{g),...,Xr,i{g),...,Xr,mr{9)) £ , 

and we will ordinarily drop in the latter the dependence on g, i.e., we will write g = {xi, Xr^mr)- 
For later purposes it will be useful to introduce the horizontal group constants of G. By the 
grading assumption on the Lie algebra, we have \Vi, Vi] = ¥2- Therefore, if e^, Cj G {ei, Cm}, 
we let 

k 

(2.14) blj =^ < [ei,ej],es >g , so that [ei,ej] = , i,j = l,...,m. 

s=l 

Consider the orthonormal basis {ei, e^, ei, e^, e,.^i, e^,™,.} of g. Using 1)2. 1|) we 
define left-invariant vector fields on G by letting 

(2.15) XjAa) = {Lg)*{ej,s) , j = l,...,r, s = l,...,mj, 

where {Lg)^ indicates the differential of Lg. As in ()2.11|) we use a special notation for the first 
two layers, and let 

(2.16) Xi{g) = {Lg)^{ei) , i = l,...,m, Ts{g) = (Lg)*(es) s = l,...,k, g£G. 

Using the Baker-Campbell-Hausdorff formula 1)2.2(1 we can express (|2.16p using the exponential 
coordinates ()2.13|) . obtaining the following lemma. 

Lemma 2.1. For each i = l,...,m, and g = {xi, ...,Xr,mr)) we have 

(2.17) = + EE 

* j=2 s = l 

d '' d 

j=2 s=l -" 
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where each bj^isa homogeneous polynomial of weighted degree j — I- In particular, if G has 
step r = 2, then for every i = 1, m, one has 

(2.18) X. = A + ix:<K..».l.^->.J; 

* s=l ^ 

d 1 AJ^., d 



- dxi ' 2 ^^^'^ dts ' 

* s=l E=l * 

where bf^ are the group constants defined by (|2.14|) . We notice that an immediate consequence 
of (ITTTl) is that 

(2.19) divE Xi = , i = l,...,m , 

where divE Xi indicates the Euclidean divergence of Xi with respect to the exponential coordi- 
nates. 

By weighted degree in the statement of Lemma l2.1l we mean that, as previously mentioned, the 
layer Vj, j = 1, ...,r, in the stratification of g is assigned the formal degree j. Correspondingly, 
each homogeneous monomial ^^2^---'^"'', with multi-indices Uj = (aj,i, ...,aj^mj),j = l, is 
said to have weighted degree k if 

r rrij 
j=l s=l 



3. Two basic models 



In this section we describe two basic models of Carnot groups. The first example is the Heisen- 
berg group with step r = 2. Such group plays an ubiquitous role in analysis and geometry, 

see e.g. (Sill, ES], IH^, IEoI]-IEo3j, [ED-IES!, [EiEI, iKokij - |EHn2] . poSj . |(^L>PTj . 

From the standpoint of geometry constitutes the central prototype of a pseudoconvex CR 
manifold, with vanishing Webster- Tanaka curvature. In fact, via the Caley transform it can be 
identified with the boundary of the Siegel upper half-space 

n 

V+ = {(2;,z„+i)GC"+i|/mz„+i > 2^|z,f}, 

i=i 

see Ch.l2 in |St2j . The second example is the cyclic, or Engel group ^, of step r = 3, see |CGrj . 
[Monj . This is an interesting example to keep in mind since it represents the basic prototype 
of a group of step r = 3, and thereby constitutes the next level of difficulty with respect to the 
Heisenberg group. Some fundamental analytical and geometric properties are true for Carnot 
groups of step r = 2, but fail for groups of step r > 3. In this respect, ^ is the simplest sub- 
Riemannian model in which to test whether conjectures which are true in step two continue to 
be valid in step three or higher. 

The Heisenberg group H". The underlying manifold of this Lie group is simply M^""*"^, with 
the non-commutative group law 

(3.1) g g = ix,y,t) {x',y',t') = {x + x' ,y + y' ,t + t' + x,y' > - < x' ,y >)) , 
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where we have let x,x',y,y' £ M", t,t' G M. Let (Lg)^ be the differential of the left-translation 
(|3.H) . A simple computation shows that 

foo\ (T \ ( ^ \ '^''f V ^ yi ^ -1 

/ 5 \ de/ a 5 

(-^)* (I) '^--li 

We note that the only non-trivial commutator is 

[Xi,Xn+j] = 5ij T , i,j = l,...,n, 

therefore the vector fields {Xi, ...,X2n} generate the Lie algebra (]„ = M^"'^-'^ = Vi © V2, where 

Vi = M^" X {0}t, V2 = {^}(x,y) X IK- We notice that the sub-Laplacian (see (jHUll)) associated 

d 

dyi ' ■ ■ ■ ' dyn 

2n 



with the orthonormal basis {-^, of Vi is 



(3.3) A„ . . A.„ + l,|.P.|.r^, ^ _ I g _ .,.|-} . 

which coincides with the real part of the complex Kohn-Spencer Laplacian, see |St2j . The 
non-isotropic group dilations are 

(3.4) 5x{g) = {Xx,Xy,Xh) , 

with homogeneous dimension Q = 2n -|- 2. A convenient renormalization of the gauge ()2.4() is 
given by 

(3.5) N{g) = {{\xf + \y\'r + 16 t')'^' . 

The importance of such function is connected with the discovery due to Folland |Flj that the 
fundamental solution of (|3.3j) is given by 

(3.6) m = T{g,e) = , 

where Cq < is an explicit constant. 

As a useful illustration, we compute the metric tensor gijd^i® d^^j associated with the smooth 
Riemannian product on with respect to which {Xi,X2,T} is an orthonormal basis. From 
1)3. 2|) we obtain 

^ ' dx 2 ' dy ' 2 ' dt 

and therefore the metric coefficients are given by 511 =< ■§^,-§^ >= 1 + ~^ ^x^^ ^~ 

— etc. One easily finds 

(1 _i_ vL —EM. y 
^ 4 4 ^ 2 
-f l + X --2 
y I 
2 2 ^ 

Notice that, since det{gij) = 1, the volume form is given by the standard (Lebesgue) volume 
form dx Ady A dt in M^. The inverse (g^^) of the matrix (|3.8|) is given by 

1 -f 

(3.9) (g^^) =(01 § 

y X 1 , x^+y^ 
2 2-^""" 4 
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Recall now the expression of the Riemannian gradient in local coordinates, see for instance 
Hi p.387, 



N 



(3.10) 



du d 



9- 



where we have denoted by = dim(G). Keeping in mind 1)3. 2jl . a simple calculation gives 



Xiu 

X2U 



Xiu 

X2U 



From this formula, and from 1)3. 7|) . (|3.9|) . we finally obtain 



(3.11) Vn = < {g'^) I , , , ^ 



d_ 



Xiu Xi + X2U X2 + TuT , 



which verifies 1)5. 15() . It is worth observing that the Laplace-Beltrami operator is given by 

An = XiXiu + X2X2U + TTu . 



The four-dimensional Engel group. We next describe the four-dimensional cyclic or Engel 
group. This group is important in many respects since it represents the next level of difficulty 
with respect to the Heisenberg group and provides an ideal framework for testing whether results 
which are true in step 2 generalize to step 3 or higher. The reader unfamiliar with the cyclic 
group can consult |CGrj . or also |Monj . The Engel group ^ = K^^, see ex. 1.1.3 in |CGrj . is the 
Lie group whose underlying manifold can be identified with M^, and whose Lie algebra is given 
by the grading, 

e = © V2 © V3 , 

where Vi = span{ei, 62}, V2 = span{e^}, and V3 = span{ei], so that mi = 2 and m2 = = 1. 
We assign the bracket relations 

(3.12) [61,62] = 63 [61,63] = 64 , 

all other brackets being assumed trivial. For the corresponding left-invariant vector fields on 
g given by Xi{g) = {Lg)^{ei), i = 1,2, T{g) = (Lg)*(63), S{g) = (Lg)*(64), we obtain the 
corresponding commutator relations 

(3.13) [Xi,X2] = T [Xi,T] = [Xi,[Xi,X2]] = S, 

all other commutators being trivial. We observe that the homogeneous dimension of C; is 

Q = mi + 2 7712 + 3 m3 = 7 . 

We will denote with {x,y), t and s respectively the variables in Vi, V2 and V3, so that any 
^ G c can be written as ^ = xei + ye2 + t63 -|- 564. If 5 = exp(^), we will identify g = (x, y, s). 
The group law in C; is given by the Baker-Campbell-Hausdorff formula ()2.2() . In exponential 
coordinates, if 5 = exp(,^), g' = exp(^'), we have 

9 o g' = i + + + ^ - [r,[e,e']]}. 

A computation based on 1)3. 12(1 gives (see also ex. 1.2.5 in |CGrj l 

g o g' = [x + x\y + y',t + t' + P^,s + s' + pA , 
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where 



{xy - yx') 



P4 = ^{xt' -tx') + ^(^x'^y' - xx'{y + y') + yx"^ 

Using the Baker-Campbeh-Hausdorff formula we find the following expressions for the vector 
fields Xi, 



(3.14) 



{X, = 

X2 = 

rp d , X d 

^ — at ^ 2 ds ' 



d_ _ y d_ _ , xy\ d_ 

dx 2 dt I2 12; as ' 

d , X d , x"^ d 

dy ^ 2 dt ^ 12 ds ' 



S 



at 
A. 

ds 



We note that the action of Xi , X2 , T on a function on (E which is independent of the variable 
s reduces to the action of the corresponding vector fields in H^. 



4. The subbundle of horizontal planes 



Consider a Carnot group G, with Lie algebra g = Vi © • • • © with an orthonormal basis 
{ei, Cm} of the horizontal layer Vi, and corresponding system X = {Xi, Xm} of generators, 
where Xi{g) = {Lg)^{ei), g £ G. Henceforth, the fiber HgG of the horizontal bundle at a point 
g & G will be denoted by Hg, so that HG = L)g,^cHg. We note explicitly that Hg = geyip{Vi), 
where exp : q ^ G denotes the exponential mapping. We will call Hg the horizontal plane 
through g. For example, when G is the Heisenberg group H", then a simple computation shows 
that the horizontal plane through a point go = {xo,yo,to) is given by the hyperplane 

(4.1) Hg = !^{x,y,t) \ t = to + ^{<xo,y> - <yo,x>)^ . 

More in general, we have the following result which is Proposition 4.3 in |DGNlj . 



Proposition 4.1. Let G be a Carnot group of step 2, then for any given gQ & G the horizontal 
plane passing through g^ is the collection of all points g £ G whose exponential coordinates 
{x,t) = {x{g),t{g)) verify the k linear equations 

^ m 

^s{g) = ts{g) - tsigo) - - ^ bfj Xi{go) Xj{g) = 0, s = l,...,k, 

*j=i 

where 6|- represent the horizontal group constants defined by ()2.14() . 



Another interesting example is provided by the four-dimensional Engel group <B described 
in the previous section. Identifying <S. with M^, with coordinates g = {x,y,t,s), given a point 
go = ixo,yo,to, So) we have that Hg^^ = span{Xi{go), X2{go)}- A simple computation based on 
1)3. 14(1 shows that Hg^^ is described by the two equations 



(4.2) 



^i{x,y,t,s) = t-to + ^^^^ = , 
^2{x,y,t,s) = S-SO+ -(6^°+-oyo)-x§y-6.oio ^ 
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From ()2.17|) in Lemma l2.ll we see that for a Carnot group G of step r, with N = dim{G), 
the horizontal plane Hg^^ is described by a system oi N — m linear equations for the exponential 
variables, see (|2.13() . 

{^i{g) = h{g) - h{go) - B^{g) = , 
^fc(5) = tk{g) - tkigo) - Bk{g) = 0, 
^r,l(5) = Xr,i{g) - Xr,i{go) - Br,i{g) = 0, 



(4.3) 



with Bj{go) = for j = 1, ...,k, ... ,Br,j{go) = 0, j = 1, ...,mr. 

Definition 4.2. We say that S <Z G is a hypersurface if S is a co-dimension one immersed 
manifold of class . If, in addition, S is embedded, then we say say that it is an embedded 
hypersurface. 

We note explicitly that, by the implicit function theorem, for every go € S there exists an 
open set O C G and a function G C'^(C') such that: (i) \V(j){g)\ / for every g € O; (ii) 
SnO = {g^O \ (/){g) = 0}. When we will need to use this local representation, we will 
always assume that S is oriented in such a way that for every go £ S and (j) as in (ii), one 
has N{go) = '^4'i9o)i where N denotes the non-unit Riemannian normal to S. The following 
notion plays a pervasive role in sub-Riemannian geometry, as well as in the study of subelliptic 
equations. 

Definition 4.3. Given a hypersurface S C G, a point go £ S is called characteristic if one 
has Hg^ C Tg^^S. Notice that this is equivalent to saying that 

(4.4) X,(5o) G Tg,S , j = l,...,m . 

The characteristic locus of 5, Tig, is the collection of all characteristic points ofS. 

Although we will not use in this paper the following two results, we recall them because of 
their interest. The first theorem is a special case of a result due to Derridj |Delj . |De2j . 

Theorem 4.4. Let S be a C°° hypersurface in a sub-Riemannian space M of dimension N , 
then denoting with the s-dimensional Hausdorff measure constructed with the Riemannian 
distance one has 

H^'-^^s) = . 

For Carnot groups one has the following sharper result first proved in codimension one by 
Balogh for the Heisenberg group 0, and subsequently extended to arbitrary Carnot groups and 
codimension by Magnani jMalj . |Ma2j . 

Theorem 4.5. Let G be a Carnot group and denote byTC^ the s-dimensional Hausdorff measure 
constructed with the Carnot- Caratheodory distance. For any manifold of codimension k one 
has 

In particular, the characteristic set of a hypersurface has zero -measure. 
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Since for a hypersurface in a Carnot group G it was proved in |DGJNlj . see also |DGJN2j . 
that the //-perimeter measure Ph{^;-), introduced in section |H1 below, is mutually absolutely 
continuous with respect to the Hausdorff measure 7{^~^, we conclude from Theorem 14.51 that 
for such domains the //-perimeter measure of the characteristic set is zero, i.e., 

(4.5) ani^s) = . 



Proposition 4.6. Let G be a Carnot group, and S <Z G he a hypersurface. If qq ^ S \ S5, 
denote by Sq = S D Hg^ . There exists a sufficiently small open neighborhood O of go such that 
SqHO is a submersed manifold of G of dimension m — 1. 

Proof. According to Definition 14. 2( there exists a neighborhood O of go such that Sr\0 = {g ^ 
O I (l){g) = 0}. Using the exponential coordinates we now introduce the function 

F : exp-^{0) C ^ j^Af-m+i defined by 



F 



def 



r, 1 ) • •• ) ^r,mr ) ) 



where the N — m functions are as in (|4.3|) . Clearly, we have F{go) 
by Jp the Jacobian matrix of F, we now claim that the hypothesis 



g M^-'^+i. Denoting 



go G S\Es 



rank Jpido) = N — m + 1 . 



Taking the claim for granted, we see that the conclusion of Proposition l4.6l immediatelv follows 
from the implicit function theorem (of course, by possibly restricting the neighborhood O), since 
the latter guarantees that, locally around go, the set So is a submersed manifold of class of 
dimension N — {N — m + 1) = m — 1. 

We now prove the claim in two special situations, namely that of a Carnot group of step 
r = 2, and that of the Engel group <B, leaving it to the interested reader to provide the (lengthy) 
details for a general Carnot group. Suppose then that G has step r = 2. Since go S, we know 
that V h4>{9o) 7^ 0. Therefore, there exists i G {!,... ,m} such that Xi(j){go) 7^ 0. Without loss 
of generality, let us assume that Xm<l}{go) 7^ 0. According to (|2.18|) we thus have 

^ k m 

(4.6) 4>xM + 2 Y.Y.^U ^j(5o) <i^tAm) + . 

/>, ^k) at go is now given by 



The Jacobian matrix of F 
( ^ 



■2 EIli bfiXiigo) 



-\Y.ZAm^Am) 1 



Xi{go 





4>t2 ■■ 




1 


.. 


. 





1 .. 


. 



\-kET=ib^xM ... -hET=lbL^^i9o) ... 1/ 

where all derivatives of (p are evaluated at go- We consider the (A; + 1) x (k + 1) minor 



Jf{9o) 



9Xm 9tl 

-iET=lbL^^i9o) 




1 



\-hET=ibL^^i9o) 



1/ 
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of the matrix Jpigo)- A careful examination of the special structure of the matrix Jpigo)-, and 
the cofactor expansion of its determinant, allow to conclude that 

det Jpigo) = (t)xm + 2 X] bimXi{go)(l>h + ••• + 2 X] bim^i(9o)(t>tf, + , 

i=l j=l 

where in the last equation we have used ()4.6() . This proves that rank Jpigo) = k + 1 = N — m+1, 
and therefore the claim follows for groups of step r = 2. 

If instead 5 C G = C; is a hypersurface in the Engel group, with qq = (xq, yo, to, sq) £ S \ S, 
then we can assume for instance that we have at go 

2 

(4.7) X2<P{go) = M9o) + YMgo) + ^Mgo) + 0. 

We consider the function F = {(j),'^i,^2) ■ ^ where ^i, i = 1,2 are as in Its 
Jacobian matrix is given by 

((l^x 'i>y (l^t 4's 

f -f 1 
Gto+xpyo _£o n 1 
12 12 ^ 

One readily sees that the 3x3 minor 

Jpigo) = 

has determinant given by X2(p{go)- From H4.7() we conclude that rank Jpigo) = 3 = N — m + 1, 
and again the claim follows. 

□ 




5. Horizontal Levi-Civita connection 



Let Gr be a Carnot group of step r. Henceforth in this paper we will assume that G is endowed 
with a left-invariant Riemannian metric < u,v >= gijU^v\ where u^v £ TG, with respect to 
which the left-invariant vector fields defined in H2.15() 

constitute an orthonormal frame for TG. No other inner product will be used on TG, thereby 
when we write < •, • > there will be no risk of confusion. We denote with by V the corresponding 
Levi-Civita connection on G. Recall that V is torsion free, 

(5.1) VxY-VyX = [X,Y] , 
and that it is metric preserving, i.e., = or, equivalently, 

(5.2) X <Y,Z> = < VxY, Z> + <Y, VxZ > . 

Permuting cyclically the roles of X, Y, Z in ()5.2|) , one obtains the basic Koszul identity, see 
e.g. (1.13) on p.28 of [Si, 

(5.3) 2 < VxY, Z> = X <Y,Z > +Y < X,Z > -Z < X,Y > 

- <Y,[X,Z]> - < X, [Y, Z]> + <Z, [X, Y] > . 
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Using H5.3|) it is easy to check that 

(5.4) Vx^Xi = 0,i = l,...,m, Vx.^m^Xj^rn, = 0,j = l,...,r. 

In addition to (|5.4() , we can easily verify from H5.3|) and the grading of the Lie algebra, that 

(5.5) <Vx,Xj,Xe > = , i,j,l = l,...,m. 

The remaining covariant derivatives and the Christoffel symbols can be determined from the 
group constants. For instance, we have the following proposition. 

Proposition 5.1. Let G be a Carnot group of step r, then 

k 

(5.6) Vx^X, = ^Y.^tjTs , i,j = l,...,m . 

^ 8 = 1 

(5.7) Vt,T, = , p,s = l,...,k . 

(5.8) Vx,Ts = --Y,b^jXj + -Y,<[X^,Ts],X3,p> X3,p , i = l,...,m, s = l,...,k. 

j=i p=i 

In particular, when G = H" one has for i,j = 1, ...,n, 

5ij 1 1 

"^XiXn+j = ^ ' "^XiT = VrXi = - - Xn+i , Vx„+,T = VrXn+i = - Xi . 

Proof. Using (|5.3|) . for any vector field 

m k r 

Z = ^ aiXi + ^ bsTs + X] CpXh^p , 

£=1 s=l /i=3p=l 

we obtain 

m k 

< Vx^Xj, Z> = ^ a£ < Vx,Xj,Xe > + ^ 6, < Vx,Xj,Ts > 
e=i s=i 

h=3 p=l 

Now (|5.5p gives < VxiXj,X£ >= 0, whereas using (|5.3|1 again, we find 

k 

2 < Vx,X,,Ts > = - <Ts, [Xj,X,] > = Y^hPjSsp = b: 



s 

p=l 



Similarly, for h G {3, ...,r} we have 

k 

2 < Vx,Xj,Xh,p >= Yl < ^h,p, T,>= . 

3 = 1 

From these equations we obtain 

k 



<Vx,x„z> = <i^6|^.r„z> 



\=i 



From the arbitrariness of Z we conclude that (|5.11|) holds. In a similar way, one obtains 
H5.12() . and ()5.13() . We leave the details to the reader. 

□ 
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Next, we want to introduce a connection on the horizontal bundle. We do this by projecting 
onto HG the Levi-Civita connection V. 

Definition 5.2. If X is a vector field on G, and Y is a horizontal vector field on G, then we 
define the (Levi-Civita) horizontal connection on HG as follows 

m 

(5.9) V^Y =^ Yl < ^xY,Xi > Xi . 

i=l 

Let us notice that V''^ satisfies the metric compatibility condition 

(5.10) X <Y,Z > = < V^Y, Z> + <Y, V^Z > , 

for every triple of vector fields X, Y, Z on G, such that Y and Z are horizontal. This follows 
from the corresponding compatibility condition ()5.2() satisfied by the Levi-Civita connection V, 
and from the definition of V^. From Proposition 15.11 and Definition 15.21 we obtain. 

Proposition 5.3. Let G be a Carnot group of step r, then 

(5.11) Vf^X, = , i,j = l,...,m . 

(5.12) V%T, = , p,s = l,...,fc . 

^ m 

(5.13) V£r. = - ^Y^^tjXj , i = l,...,m, s = l,...,k . 

i=i 

Remark 5.4. We mention that the horizontal Levi-Civita connection is intimately con- 

nected with the notion of non-holonomic connection introduced by Cartan in his address at the 
1928 International Congress of Mathematicians in Bologna ^C_. In this respect we refer the 
reader to the interesting re-visitation of Cartan's address by Koiller, Rodrigues and Pitanga, see 
|KKPlj ■ |KRP2j ■ where the authors generalize some of the ideas in [C] and also introduce a 
non-holonomic connection (see their Definition 1.1 in |KKPlj } which, for a Carnot group, gives 
precisely our Definition 15. M A general framework has been recently set forth by Hladky and 
Pauls in |HPj for what they call vertically rigid spaces. These are sub-Riemannian manifolds 
which include, in particular, Carnot groups. When specialized to Carnot groups, the adapted 
connection in |HPj coincides with the horizontal connection in De{inition \5.'A 

Hereafter, for a given vector field X we indicate with X^ = YllLi < X,Xi > Xi the projection 
of X on the horizontal bundle HG. 

Proposition 5.5. Given horizontal vector fields X and Y , one has 

•m 

V^Y - V^X = [X,Yf =^ Yl < i^^nXi > Xi . 

i=l 

Proof. From Definition l5.2l and the torsion freeness 1)5. If) of the Levi-Civita connection we obtain 

m 

V^Y - V^X = Y.< - > 

i=l 
m 

i=l 



SUB-RIEMANNIAN CALCULUS ON HYPERSURFACES IN CARNOT GROUPS 



19 



□ 

If we define the horizontal torsion as follows 

T^{X,Y) = V^Y - V§X - [X,Y]" , 

then Proposition 15.51 asserts that the horizontal connection is torsion free, and this is why we 
call it the horizontal Levi-Civita connection. Permuting cyclically the roles of X,Y,Z in (|5.10() . 
and using Proposition 15. 5( we obtain the following horizontal Koszul identity for V^. 

Proposition 5.6. Let X, Y, Z be horizontal vector fields on G, then 

(5.14) 2 < y, Z> = X <Y,Z > +Y < X,Z > -Z < X,Y > 

- <Y, [X, Z]" > - < X, [Y, Z]" > + < Z, [X, Y]^ > . 

ProDosition l5.6l shows in particular that is completely determined by the Riemannian inner 
product in G and by the horizontal bundle HG. Given a function u G C^(G), its Riemannian 
gradient with respect to the inner product < •, • > is given by 

(5.15) Vu = XiU Xi + ...+XmU Xm + TiuTi + ... + TkUTk + ...+Xr^lU Xr^l + ...+Xr^mrU Xr^rur ■, 

If we let G = det{gij), then as a consequence of 1)2. 19() . and of the fact that G = 1 (see |F2j . 
or |CGrj l. we obtain for the divergence of Xi (see |Hej p. 387) 

(5.16) div Xi = —Y,j^ (VG{X,)k) = divEX, + ^(Xi),. — (logVG) = , 



for every i = 1, m. The horizontal gradient of u is obtained by projecting Vn on the subbundle 
HG (see Definition 15. 2 j) . The resulting horizontal vector field on G is nothing but the horizontal 
connection acting on u 

(5.17) V^U = < Vu,Xi > Xi + ...+ < \/u,Xm > Xm = XiU Xi + ... + XmU X-m ■ 

If C = Ci-'^^i + ••• + CmXm S C^{G, HG), then the horizontal divergence of ( is given by 

(5.18) divnC = XiCi + ... + XmCm ■ 

The horizontal Laplacian (also known as sub-Laplacian) of a function u G C'^iG) is given by 

m 

(5.19) Aj/ii = divH^^u = ^Xf . 

i=l 

Except for the Abelian case when the step r = 1 and A/^ is just the standard Laplacian 
A = "^ZlLid"^ /dxj, such operator fails to be elliptic at every point of G. We notice that 
A//n = traceCVjju), where we have denoted by V^u the m x m matrix- valued function on G 
defined by 

fKnn\ V72 XiXjU + XjXiU . . 

(5.20) Vfju = u^ij = '—^ — , i,j = l,...,m . 

The following proposition contains a useful property of Carnot groups. 

Proposition 5.7. Let G be a Carnot group, then 

(5.21) XiXj = 6ij , AnXj = 0, i,j = l,...,m. 

As a consequence, we find 

(5.22) |V^(|x|2)|2 = 4 Ixp . 
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One also has 

1 1 ^ 1 

(5.23) X,ts = - < [^i,ei],es > = - ^x,b%, X,X,t, = - h]^ . 

i=i 

In particular, we obtain V^(ts) = 0, and therefore Antg = 0, s = 1, k. 



6. Horizontal Gauss map and tangent space to a hypersurface 



In this section we introduce two basic geometric concepts for an hypersurface in a Carnot 
group G which are adapted to the horizontal subbundle of G. We consider the Riemann- 
ian manifold M = G with the metric tensor with respect to which Xi, X^, Xr^mr is an 
orthonormal basis, the corresponding Levi-Civita connection V on G, and the horizontal Levi- 
Civita connection introduced in Definition 15.21 Let 5 C G be a oriented hypersurface, 
with k >2. We will denote by N the non-unit Riemannian normal to 5, and will indicate with 
V = N /\N\ the Riemannian Gauss map of S. It will be convenient to introduce the following 
notation 



(6.1) pj = <N,Xj > , i = j,...,m , W = ^pI + ... + p^ 
We now set 

(6.2) p. = El ^ so that pI + ... + pI, = 1 on 5 \ 
We also define 

(6.3) LUs = < N,Ts > , lJs = ^ , s = 1, k , 



W 



ujj,s =<N,Xj^s>, ujj,s = '-^ , j = l,...,r, s = l,...,mj. 

go £ S is characteristic, then we have Pj{go) = 0, j = 1, ...,m, and therefore we have the 
alternative characterization of as the zero set of the continuous function W 

(6.4) = {geS\ W{g) = 0} , 

which shows that is a closed subset of S. The next definition plays a basic role in the sequel. 
Definition 6.1. We define the horizontal normal : S HG by the formula 

m m 

(6.5) = Y^< N,X, > X, = Y,Pj X, . 

i=i i=i 
The horizontal Gauss map is defined by 

(6.6) = —jj- = Y,Pj , on S\^s . 

I I j=i 

We note that is the projection of the Riemannian normal N on the horizontal subbundle 
HG C TG. Such projection vanishes only at characteristic points, and this is why the horizontal 
Gauss map is not defined on A trivial consequence of the definition which, however, will be 
important in the sequel is 

(6.7) = pI + ... + = I ^ in S\^s , 
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which is of course a re- formulation of the second equation in (|6.2() . One also has 

(6.8) < v^,N^ > = \N^\ , - < N^,u^ > = . 

We note explicitly that, with these quantities in place, the Riemannian (non-unit) normal to 
S is given at every g £ S \ T,s i>y 

(6.9) N = + UJiTi + ... + UJkTk + ...+UJr,mrXr,mr 

= wi^PiXi + ... +Pj^Xm + WlTi + ... +UJkTk + ... +UJr,mrXr,mr 



Since < ,Ts >=< ,Xj^mj >= for s = 1, ...,k, and j = 3, ...,r, it is obvious from (|6.9() 
that 

(6.10) < N,u" > = < > = W, hence cos{v^ AN) = 

Because of (|6.10|) , the function W is also called the angle function. 

Remark 6.2. To help the reader's comprehension, we sometimes give proofs or examples in 
the special case when G = , the first Heisenberg group. Furthermore, Sections MiA and \15\ 
are devoted to this special setting. It will thus he convenient to simplify the notation introduced 
above as follows. For surfaces S CM^ we will let 



(6.11) P = Pi , q = P2 , u = uji , W = \Ip^ + g2 , p = , g, = p2 , a; = wi . 

Consequently, in this setting the normal N and the horizontal Gauss map will always be 
respectively written as 

(6.12) N = pXi + qX2 + ujT = N" + ujT, = p Xi + q X2 , 
so that ()6.9|) becomes 

N = W{u^ + iJT} . 
The horizontal vector field defined on S \ T,s by 

(6.13) (u^)^ = qXi - pX2 , 

is perpendicular to , but it is also orthogonal to the Riemannian normal N to S. 



Definition 6.3. At a point g £ S \ the horizontal tangent space is defined as follows 

HTgS =^ {V £Hg\<V,V^ >g = O} . 

The horizontal tangent bundle of S is defined by 

HTS = IJ HTgS . 

One can check that HTS has the structure of a vector bundle. It is clear that, since dim Hg = 
m, then dim HTgS = m — 1, and one has in fact 

(6.14) Hg = HTgS © span {i^^ {g)} . 

For instance, when G = M.^, then if for a surface 5 C H"'^ we consider the unit vector field 
on S given by ()6.13|1 . then it is clear that at every point g £ S \ Ti, one has 

(6.15) HTgS = span{{i^'')^ig)} . 
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If we consider F = 5 H Hg, then from Proposition 14.61 we know that F is submersed manifold 
of dimension one (a curve). Its Riemannian tangent space in g can be identified in a canonical 
way with HTg. We also observe that an orthonormal basis for the Riemannian tangent space 
TgS oi S a,t g is given by 

(6.16) TgS = spanl^iu^)^ , ^..^ - . 
Proposition 6.4. Let g £ S \ E^, then one has 

HTgS = TgS n Hg . 

Proof. We begin by observing that, since by hypothesis g S^, then Hg <f_ TgS, and therefore 
7^ 0. Now, from (|6.9|) and the fact that Xi, Xm,Ti, ...,Xr^rnr constitute an orthonormal 
basis of TgS at every g G G, one sees from (inSI) that N - _L Hg. Therefore, if v £ HTgS, 
then we have 

(6.17) <v,N > = <v,N - > + <v,N^>= 0, 

which shows v G TgS. We thus have the inclusion HTgS C TgS D Hg. To establish the opposite 
inclusion, let v £ TgS n Hg. We thus have that the left-hand side of 1)6. 17() is zero, and since 
< V, N — N^ >= because of the fact that v G Hg, we conclude that it must he < v, >= 0, 
hence v G HTgS. 

□ 



7. Horizontal connection on a hypersurface 



We recall the classical definition of the Levi-Civita connection of a n-dimensional immersed 
submanifold = A'^" of an m-dimensional Riemannian manifold M = M™'. Denoting with 
i : N ^ M the immersion, and having endowed N with the induced Riemannian metric i*g, 
let : TN TM be the differential of i. We identify TpN with the subspace (i*)p(TpA^) of 
TpM, and denote by TpN-^ its orthogonal complement. TN-^ = UpeAr TpN-^ has the structure 
of a (m — n)-dimensional vector bundle, traditionally referred to as the normal bundle of N . 
We can thus write TM \ N = TN © TN-^, and for every u G TpM, we indicate with its 
TpN component, and with its TpN-^ component. Since p — > (V;^y)~'^(p) satisfies all the 
assumptions of a Levi-Civita connection on A^, by the uniqueness of the latter we obtain 

(7.1) y = {vj^yy . 

Before proceeding we need to say a few words concerning 1)7.11) . First of all, since X, Y are 
only initially defined on the submanifold A^, we need to give a meaning to right-hand side. 
Using a partition of unity argument, we can extend X, Y to smooth vector fields X, Y on M, 
and therefore interpret the right-hand side as follows 

(7.2) (vf y)^ = (VfF)T . 

This immediately raises the question of whether (|7.2() is a good definition, in other words, 
whether it is independent of the particular extensions of X, Y that we have picked. Since the 
value of Y at p £ N depends only on Xp, it is clear that 1)7. 2() is independent of the extension 
of X. On the other hand, (V^y)p depends only on the values of Y along any curve on M whose 
initial tangent vector is Xp. By picking a curve which lies entirely on A^, we see that ()7.2|) is 
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also independent of the extension of Y. This fact, can be also recognized by the following 
observations, which also establish the torsion freeness of the connection (V^y)^. Denoting 
with the differential of the immersion, we have = X, = Y, and therefore, see 

Theorem 7.9 in |Boj . y] = [X,y]. This implies, in particular, that y] = [X,y]'''. From 
the torsion freeness of V^'^, we thus conclude that 

(7.3) (Vfy)^ - {vMxy = [X,y]T = [X,Y] . 

We notice that [X, Y]p only depends on the values of X, y in a neighborhood of p in A^, and 
therefore (|7.Hj) shows at once that (|7.2j) is a good definition, and that (V^y)"'' is torsion free. 
The remaining properties of a Levi-Civita connection are checked easily. 

Inspired by the Riemannian situation we now introduce a notion of horizontal connection on 
a hypersurface 5 C G by projecting the horizontal Levi-Civita connection in the ambient 
Lie group G onto the horizontal tangent space HTS. 

Definition 7.1. Let Let S C G be a non-characteristic, hypersurface, k>2, then we define 
the horizontal connection on S as follows. LetX/n denote the horizontal Levi-Civita connection 
introduced in Definition \5.iA For every X,Y ^ C^{S; HTS) we define 

v^Y = vf y - < vf y, > , 

where X, Y are any two horizontal vector fields on G such that X = X,Y = YonS. 

Arguing as above one can check that Definition 17. II is well-posed, i.e., it is independent of the 
extensions X, y of the vector fields X,Y. From Proposition 15.51 we immediately obtain. 

Proposition 7.2. For every X, y G C^{S; HTS) one has 

Vf'^y - Vy'^X = [X,Yf - < [X,Y]^,u^ > . 

It is clear from this proposition that the horizontal connection V^'*^ on S is not necessarily 
torsion free. This depends on the fact that it is not true in general that, if X, y E C^{S] HTS), 
then [X,Y]^ G C\S;HTS). In the special case of the first Heisenberg group this fact is true, 
and we have the following result. 

Proposition 7.3. Given a non- characteristic surface S C , k > 2, one has [X,Y]^ G 
HTS for every X,Y £ C^{S; HTS), and therefore the horizontal connection on S is torsion 
free. 

Proof. According to 1)6. 15() . for every g S we have HTgS = span{ei{g)}, where ei = (iv^)-*-. 
Therefore, if we take two vector fields X, y G C^{S; HTS), then we can write X = aei, Y = bei, 
for appropriate C^~'^ functions a and b. We thus have 

[X, y] = [aei,bei] = {aei(6) — 6ei(a)} ei 

This shows that [X, y] G C{S; HTS), and therefore Proposition 17.21 gives 

V^'^Y - Vy'^X = [X,Y] . 

This gives the desired conclusion. 

□ 
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Definition 7.4. Let S be as in Definition \7.1\ Consider a function u G C'^{S). We define the 
tangential horizontal gradient of u as follows 

where u G C^{G) is such that u = u on S. 

We note that V^^^u = YT=i ^i'^u Xi, where 

Vf "^u = Xiu- < V^u, > zyf = XiU - p, pj XjU . 
Observe also that V^''^u G HTS. One has in fact from 1)6 .71) and Definition 17.41 

(7.4) < V^"^u, i/^ > = in 5 \ S , 
and therefore 

(7.5) |V^"5n|2 = \V^u\^- <vV«-^>' . 



8. Perimeter measure and horizontal first fundamental form 

In a Carnot group G, given an open set C G, we let 

m ^ m X 1/2 

= {C = y^CiX,eCl{n,HG) I iCloo = sup Vcf <!}• 
For a function u G Lj^^{Q), the ff- variation of u with respect to is defined by 

VarH{u;Q) = sup / u diYnC dg . 
CeTin) Jg 

We say that u G L^(r2) has bounded //-variation in if VarH{u; 17) < oo. The space BVh{^) 
of functions with bounded //-variation in fi, endowed with the norm 

ll^^llByH(!^) = ll^llLi(n) + VarHiu;^) , 

is a Banach space. 

Definition 8.1. Let E C G be a measurable set, Q be an open set. The H -perimeter of E with 
respect to Vt is defined by 

PH{E;n) = VarHixE;^) , 
where xe denotes the indicator function of E. We say that E is a H-Caccioppoli set if xe £ 
BVni^) for every Q CC G. 

The above definitions are taken from |CDGj . see also |GNj . Following classical arguments [Z]. 
|EGj . one obtains from the Riesz representation theorem. 

Theorem 8.2. Given an open set G G, let E C G be a H-Caccioppoli set in Q. There exist 
a Radon measure \ \d^ E\\ in Q., and a \ \d^ E\ \ -measurable function : Q — > HG, such that 

\u^{g)\ = 1 for P^/^ll -a.e. gen, 

and for which one has for every C G Cq^^I; HG) 

[ divuCdg = [ <C,i^E> d\\d"E\\ = [ <C,d[d"E]> . 
Je Jn Jn 
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Let E C G he a domain, with Riemannian outer unit normal u. If C £ Cq{0,] HG), we 
have 

» ,. m 

/ div H C dg = S2 '^i < ^i^'^ > dHN-i ■ 

Je JdEnn ~l 

From this observation, and from Theorem 18.21 we conclude the following result. 

Proposition 8.3. Let E C G be a domain. For every open set Q C G, and any G 
Cq{Q.;HG), one has 

r r ]\jH 

/ <C,iy^> dWd'^EW = / <C,^> dHM-1, 

JQ JdEnn M^l 

where is defined in 1)6. 5|) . Moreover, 

(8.1) d\\d^E\\ = \N"\ d{HN-ildE) , 
and one has 

u f \N^\ r W 

(8.2) Wd^'Em) = PniE-n) = / dHr,-i = 

JdEnn M^l Jd 

where W is the angle function defined in 



ATI 

dEnn \-^^ \ JdEnn M^l 



Definition 8.4. Given an oriented C hypersurface S C G, we will denote by 

\N^\ W 
(8.3) daH = dHN^ilS = ^ dHN-i[S , 

the H-perimeter measure supported on S (see 1)8.2(1 and ()6.1U|) ). 

For a detailed local study of such measure the reader should see IDCiiNlj . |D(TN2j . |Malj . 
|Ma2j . An interesting interpretation of the //-perimeter measure is that the latter is obtained 
by blowing-up the Riemannian regularizations of the sub- Riemannian metric of the group G. In 
a different context, this idea was first exploited systematically by Koranyi |Kolj in his compu- 
tations of the sub-Riemannian geodesies in H". For simplicity, and to illustrate the main idea, 
we will state the relevant result in the case when G = H". 



Theorem 8.5. Consider in the Heisenberg group H" the left- invariant Riemannian metric ten- 
sor {5ij}i,j=i,...,2n+i with respect to which {Xi, X2n, \/eT} constitutes an orthonormal frame 
of TEI". Let S C H" be a hypersurface, with E5 = 0, and denote by lf{-, •) the first fun- 
damental form in the Riemannian metric on S induced by {gfj}ij=i,...,2n+i- Denote by the 
corresponding surface area on S, then for any bounded open chart U C S one has 

Proof. For simplicity, we present the proof in the case n = 1. Let = \/eT, and consider in 
the one-parameter family of left-invariant Riemannian metrics {(fl'ij)ij=i,2,3}e>o with respect to 
which {Xi,X2,Te} constitute an orthonormal basis of T'M} . Similarly to ()3.8() . we find 

(8.4) (4) = 

One easily verifies that 



4e 4e 2e 

_xy I _x_ 

\ JL 1 / 

\ 2e 2e e / 

= det{gl.) = 
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and that letting {{g^Y^) = (5^,)"^, then 




(8.6) 



^•5) {{9 

\~ 2 2 ' 4 

Let C be a bounded open set such that U is represented hy 9 : ^ U , with G C^(r2). 
We have 6{u,v) = x{u,v)Xi + y{u,v)X2 + t{u,v)T, see (|14.15() . We now use some of the 
computations from Section ITU We rewrite (|14.17j) as follows 

'e^ = + y^X2 + ^ (t« + ^^^^) , 

= x,X, + y,X2 + ^ (t. + ^^^) . 

Denoting by the wedge product with respect to the orthonormal frame {Xi, X2,Tf^}, sim- 
ilarly to 1)14. 19() we obtain for the non-unit Riemannian normal to S with respect to •) 

(8.7) A/"' = 9u A, 9^ = -^[vutv-yvtu - ^{xuVv - Xvyu)^ Xi 

1 / X \\ V 

~\~ "^J^ y^v^u ^utv ~l" '^{■^uVv XvlJu) J X2 

= -^pXi + -^qX2 + ujT^ , 
where in the last equality we have used ()14.20() . From (|8.7() we conclude that 

(8.8) ^4=^ = f da, = = ^fe f ^I,{N',N')du Adv = [ ^fv^^V^^VeJ^ duAdv . 
Letting e ^ in we conclude 

lim^^ = I W duAdv = I ^da = (7h(U) , 
where in the last equality we have used (|8.3j) . This completes the proof. 

□ 

We close this section by collecting two basic properties of the i/-perimeter. The former is a 
trivial consequence of the left-invariance on the vector fields Xi, X^, and of the definition of 
iif-perimeter. 

Proposition 8.6. For any H-Caccioppoli set E in a Carnot group G, and any open set d G, 
one has 

(8.9) PH{Lg,{E)-Lg,{^)) = Ph{E-^) , goeG, 
where Lg^g = g^g is the left-translation on the group. In particular, 

(8.10) PHiLg,{Ey,G) = Ph{E;G) , go & G . 

Proposition 8.7. In a Carnot group G one has for every H-Caccioppoli set E C G, any open 
set Q, C G, and every A > 

(8.11) PH{6xE;6xn) = X^-^ Ph{E;^) • 
In particular, 

(8.12) Ph{5xE;G) = A^'^ Ph{E-,^) ■ 
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Proof. We observe that if C G C^{G, HG), then G Cl{5x^; HQ). Furthermore, C G 

if and only if o Sijx G J-{5Vt). The divergence theorem, and a rescahng now give 

(8.13) / <iit;j^C^^5= / Vx,0(i5 = A-« / V X,0(<^i/a5) ^5 ■ 

Since 

we conclude from H8.13() 

/ divuQdg = A-(«-i) / y^Xj{Cjo8yx)dg . 

JE JSxE -^^ 

Taking the supremum on all C, G in the latter equation, we reach the desired conclusion. 

□ 

Combining Propositions 18.61 and 18.71 we obtain the following result. 

Corollary 8.8. Let S C G be a C'^ hypersurface with finite H -perimeter, then for every go G G, 
and every A > 0, one has 



9. Horizontal second fundamental form and mean curvature 

We open this section by computing the first variation of the ff-perimeter for deformations of 
a hypersurface S along the Riemannian normal AT" to 5. This will provide a first motivation for 
the introduction of the notion of H-mean curvature. 

Theorem 9.1. Let U C G be a bounded open set and consider (j) G C'^ifJ) with |V0| > a > 
in lA, and for small A G [— Aq, Aq] consider the one-parameter family of S'^ = dlA\, where we 
have let U\ = {g ^ U \ (j){g) < A}. Assume that each of the be a non- characteristic 
hypersurface. Let S = and define a function TL : S ^M. by letting 

m 

(9.1) n=^Y.^^P^^ 

1=1 

where the p^ are the components of the horizontal Gauss map introduced in ()6.2() . We then have 
d_ 
dX 



Ph{S') 4^Ph{IIx;G) = [ 

A=0 dX A=0 



(iff AT- 1 , 



\N 



In particular, S is a critical point of the H -perimeter with respect to the deformations S ^ 
if and only of7i = 0. 



Proof. Using Federer's coarea formula jFej we can write 
(9.2) 



/ \V^cl)\dg = [ [ ^dHN-idT = [ PH{Ur;G)dT, 
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where the second equahty is a consequence of (|8.2() . The identity H9.2|) gives 

(9.3) Ph{Ux;G) = ^ ! W<t^\dg. 

dX Jux 

Using the summation convention over repeated indices, and integration by parts, we now 
compute 

(9.4) / |V^<A| dg = f Xi<t> dg 



I (/) < Xi,u > uf dHN-i - / </> Xiuf dg 
[ <l) \N''\dHN-i - / Xiv^ dg 



IdUx JUx 

= \Ph{Ux;G) - [ <pXiiyf dg , 

JUx 

where we have used (|8.2p . From ()9.3(1 . 1)9. 4[) . and the coarea formula again, we find 
(9.5) PH{Uy,G) = ±{XPH{Uy,G)} - X [ ^dH^.i . 

Equation H9.5p easily implies the desired conclusion. 



□ 



Remark 9.2. As we will see in this section, the critical points of the H-perimeter are precisely 
the so-called H -minimal hyper surf aces. We will return to this question in Section \f^\ where we 
will develop more precise intrinsic first and second variation formulas of the H-perimeter in the 
setting of the Heisenherg group. 

We are now ready to introduce the central notions of sub-Riemannian, or horizontal second 
fundamental form, and of i/-mean curvature. According to Theorem l9.11 hypersurfaces for which 
the function 7i in (|9.1|) vanishes identically on S are critical points of the -perimeter functional 
with respect to deformations of the surface in the direction of the Riemannian normal N to S. 
This suggests a notion of horizontal mean curvature of S based on the equation 1)9. Such 
notion was proposed by one of us back in 1997, see |Glj . and it produces precisely the function 
in H9.1|) . We will in fact introduce a more intrinsic notion which is based on that of horizontal 
second fundamental form, and then recognize that such definition coincides with 1)9. This 
closely parallels the classical development of the subject. We recall the classical definition of 
the mean curvature H oi a n-dimensional immersed submanifold N = of an m-dimensional 
Riemannian manifold M = M"^. Denoting with i : N ^ M the immersion, we recall that the 
Levi-Civita connection of N is given by the equation 1)7. The second fundamental form of 
is defined by 

iiNix,Y) = (vfy)^ , 

where is the Levi-Civita connection of A/. Since for vector fields on one has 

IlN{X,Y)-IIr,{Y,X) = (Vfy-V^^X)^ = [X,Y]^ = 0, 

IIj\f defines a symmetric tensor field on A^ of type (0, 2), which takes values in TN-^. The mean 
curvature of A^ at a point p £ N is defined by 



1 1 

H = trace{IlN) = //Ar(ei, Cj) 



n n 

1= 
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where {ei, e„} is an orthonormal basis of TpN . 

We now consider the Riemannian manifold M = G with the metric tensor with respect to 
which Xi, ...,Xm, ■.■,Xr, 

rrir is &I1 orthonormal basis, and the corresponding Levi-Civita connec- 
tion V on G. Let denote the horizontal Levi-Civita connection introduced in Definition 15 .21 
Let 5 C G be a hypersurface. Inspired by the Riemannian situation we introduce a notion 
of horizontal second fundamental on S as follows. 

Definition 9.3. Let S <Z G he a hypersurface, with Tig = 0, then we define a tensor field 
of type (0, 2) on HTS, as follows: for every X,Y e C^{S; HTS) 

(9.6) Il"'^{X,Y) = < V^Y,v^ > . 

We call ■) the horizontal second fundamental form of S . We also define A^'^ : HTS —>■ 

HTS by letting for every g & S and u,v £ HTg 

(9.7) <A^'^u,v>= - < II^'^{u,v),u^ > = - <V^Y,u^>, 

where X,Y £ C^{S , HTS) are such that Xg = u, Yg = v. We call the endomorphism A^'^ : 
HTgS HTgS the horizontal shape operator. If ei, e^-i denotes a local orthonormal frame 
for HTS, then the matrix of the horizontal shape operator with respect to the basis ei, e^-i 
is given by the (m — 1) x (m — 1) matrix [— < V^_ej,u^ > ^-i' 

Using the horizontal Koszul identity (|5.14j) . one easily verifies that 

Using Proposition 15.51 in Definition 19.31 we immediately recognize that 

(9.8) II^'^{X,Y) - II^'^{Y,X) = < [X,Yf,u^ > , 

and therefore, unlike its Riemannian counterpart, the horizontal second fundamental form of 
S is not necessarily symmetric. This depends on the fact, already observed, that li X,Y £ 
C^{S; HTS), then it is not necessarily true that [X, y]^ E C{S; HTS). The next proposition 
gives a necessary and sufficient condition for the symmetry of 11^'^ . 

Proposition 9.4. The horizontal second fundamental form II^'^{-,-) is a (0,2) symmetric 
tensor field on HTS if and only if for any local orthonormal basis ei, ...,6^-1 of HTS, one has 

[ei,ej]^ G HTS , i, j = 1, ...,m - 1 . 
Proof. Let X = J2'i=[^ ^.i^i, Y = YJj=i bjej, then 

m— 1 ( m—1 ^ m—1 

[X,Y] = ^ <'^{aiei{bj)-biei{aj))>ej + ^ 0,6^ [e^, e^] . 

j=l K. i=l ) *ii=l 

This identity gives 

m—1 m—1 

< [X,Y]^,v" > = ^ [aiei{bj) - biei{aj)] < ej,u^ > + Yl ^^^i < [^i^ej]" > 

i,j = l i,j=l 
m—1 

= Oibj < [ei,ej]^,u" > = , 

provided that [ei,ej]^ G HTS. Therefore, under the assumption [ei,ej]^ G HTS, we finally 
obtain from Definition 19.31 

II^'^{X,Y) - II^'^{Y,X) = <V^Y -V^X,v" >v" = <[X,Yf,-u^ >-u" = 0, 
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which proves the symmetry of 11^'^ . Vice- versa, suppose that 11^''^ be symmetric, then apply- 
ing the latter identity with X = ei, Y = ej, we reach the conclusion that < [ei,ej]^ >= 0. 

□ 

Corollary 9.5. The horizontal second fundamental form of a non- characteristic surface 
5 C is symmetric. 

Proof. In this situation the assumption of Proposition 19.41 is trivially satisfied since a basis of 
HTS is given by the single vector field ei = (i/^)-*-, and therefore [ei,ei]^ = E HTS, see 
also Proposition 17.31 

□ 

Another situation in which the assumption of Proposition 19.41 is fulfilled is that when 5 is a 
cylindrical hypersurface over the first layer of the Lie algebra. 

Proposition 9.6. Suppose that the hypersurface S is a vertical cylinder, i.e., it can be repre- 
sented in the form 

(9.9) S = {geG\t){x^{g),...,Xm{g))=0} , 

where I) G C'^{W^), and there exist an open set lo C M"* and a > such that |V{)| > a in 
LU. Under these assumptions, we have = 0, and the horizontal second fundamental form is 
symmetric. 

Proof. The function (j)(g) = Xm(5)) is a defining function of S. Using the global 

exponential coordinates, we obtain from Lemma l2.ll 

^■*<»> = £ ■ 

hence V^(p = V^f}, which proves in particular that = 0, and that = = u. We next 

observe that for every g^ € G, the left-translated surface S = Lg^^{S) is again a vertical cylinder, 
with defining function [^(x) = \){x{gQ)+x{g)) = 0. As a consequence of this observation, lig^ E 5, 
then by left-translation we can assume without restriction that g^ = e. We thus immediately see 
that the horizontal plane He = exp(yi) is given by ti = ... = tfc = ••• — Xr,mr — 0- Furthermore, 
by an orthogonal transformation in the horizontal layer of the Lie algebra, we can assume that the 
tangent space of 5 in e be given by the hyperplane Xm = 0. Since thanks to Proposition 16.41 the 
horizontal tangent space at e is given by H^, D T(,S, from the previous considerations we see that 
HTeS = spanjei, Cm-i}, where = {d/dxi)e- Since [d/dxi,d/dxj] = 0, i,j = l,...,m — 1, 
we conclude that [cj, ej]^ = 0. In view of Proposition 19.41 we conclude that 11^'^ is symmetric, 
thus completing the proof. 

□ 

From the proof of Proposition 19.61 one easily obtains the following corollary. 

Corollary 9.7. Let S he a vertical cylinder as in (|9.9|1 . then the H-mean curvature at g £ S is 
given by the formula 

(9.10) n{g) = (m-1) H{x{g)) , 

where H{x{g)) represents the Riemannian mean curvature of the projection 7ry^(5) ofS onto the 
horizontal layer Vi. In particular, S is H -minimal if and only if ttvi{S) is a classical minimal 
surface in V\ ~ . 
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Definition 9.8. We define the horizontal principal curvatures as the real eigenvalues ki, Hm-i 
of the symmetrized operator 

•^sym — 2 \ ^ ^ / ' 

The H-mean curvature of S at a non- characteristic point qq £ S is defined as 

m—l m—1 

n = - trace A^^i = ^ = ^<V^e„i/^> . 

i=l 1=1 

If 90 is characteristic, then we let 

n{go) = lim n{g) , 

provided that such limit exists, finite or infinite. We do not define the H-mean curvature at 
those points go G at which the limit does not exist. Finally, we call 7i = 7iu^ the H-mean 
curvature vector. 



Proposition 9.9. The H-mean curvature in Definition \y.8\ coincides with the function defined 
in H9.1|) . In fact, the following intrinsic identity holds 

m m 

(9.11) n = Y. vf'" <u^,x,>=Y: vf'^P.. 

1=1 i=l 

Proof. In what follows to simplify the exposition we continue to indicate with Pi,...,Prn 
m-tuple of extensions to the whole of G of the coefficients of the horizontal Gauss map with 
respect to the basis Xi, X^. We begin by observing that using the horizontal Koszul identity 
H5.14() . one easily recognizes that 

From Definition 19.81 we thus obtain 

m— 1 m—l 

(9.12) n = - 5^ < V^e,,!/^ > = <e„[e„i.^]^ > . 

i=l i=l 

Recalling 1)6. 6|) . we find 

m m 

i=i i=i 
To proceed in the calculations, we write 

m 

Si = afXj , i = l,...,m-l, 

1=1 

with {a^} satisfying the orthogonality conditions 

m m 

(9.13) ^a-p^ = 0, ^a-aj = 5ij , ij = l,...,m-l. 

1=1 1=1 
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We thus obtain 

m mm 

j=i 1=1 j=i 

k 1 m m \ 
s=l \£=lj=l / 

The latter identity gives 

m mm 
j=l 1=1 3=1 

and therefore, 

m—l m m—1 m m—1 

(9.14) < [^-^''I'^'e^ > = E E -EE ^'^M) 

i=l j=l i=l e,j=l 1=1 

m m—1 

= EE^^(^j>* 

j=i i=i 

m m—1 

= E ^^(^i) E ' 
i,j=i i=i 

where in the second to the last equality we have used ()9.13p . We now observe that, since 
{ei, Bm-i, v^} is an orthonormal basis of HgTS, we have 

m—1 m—1 

44 = ^ < X£, Si >< Xj,ej > 

i=l i=l 

= < Xg,Xj > - < Xg,u^ >< Xj,u^ > = 6ej - p(,pj . 
Substituting this identity in (|9.14|) . and recalling (|9.12j) . we finally have 

m—1 mm m 

1=1 3=1 1,3=1 j=l 

This concludes the proof. 

□ 

It is clear from Definition Ol that H £ C{S \ S). 

Definition 9.10. An oriented hypersurface S C G, k > 2, is said to have constant ff-mean 
curvature ifTC = const on S. We say that S is i7 -minimal if its H-mean curvature TC vanishes 
everywhere as a continuous function on S. 

Remark 9.11. Consider the product group G = G x M. with the canonical group law {g,s) o 
{g',s') = {gg',s + s'), induced by the one on G. The stratification of the Lie algebra for G is 
then given by Vi® ■ ■ ■ ®Vr where Vi = Vi x M, Vj = Vj x {0} for j = 2, 3, r. // {ei, Cm} 
is a basis for V\, we let Cj = {ej, 0) for j = 1, m, im+i = (0, 0, 1). A basis for Vi is then 
given by {ei, Cm+i}- This identifies a subbundle HG. We can naturally identify G with the 
hypersurface S = G x {0} C G with global defining function (j){g^s) = s. Now observe that 
Vj ' (/) = Xicj), i = 1, ...,m and V^^j^^ = 1 and therefore, = im+i- As a consequence, we 



SUB-RIEMANNIAN CALCULUS ON HYPERSURFACES IN CARNOT GROUPS 



33 



have V^'*^ = for i = 1, m + 1. In view of Provosition \y.iA we conclude that the H-mean 
curvature of G in G is zero. 

To state the next proposition we consider for a function u : G the symmetrized horizontal 
Hessian oi u at g £ G. This is the m x m matrix with entries 

(9.15) u^ij ^ jxjXjn + XjXjuj , i,j = l,...,m . 

Setting V^n = [u,ij], the mapping g \/'jju{g) defines a 2-covariant tensor on the subbundle 
HG. We recah that the horizontal Laplacian associated with the basis {ei, Cm} of Vi is given 
by Ahu = tr V'jj u. We also consider the following nonlinear operator 

m 

(9.16) Ah,ooU ^= Yl ^.ii = 2 ^ V^{\V"u\^),V"u > , 

which, by analogy with its by now classical Euclidean ancestor, we call the horizontal oo- 
Laplacian. The reason for introducing the operator Ah,oo is in the following result which is 
often useful in computing the H-mean curvature. We consider a hypersurface S C G, and 
for a given go G 5 \ S^, suppose that there exist a neighborhood U of go, and (j) G C'^iU), such 
that S f\U = 5 n d{g G U \ (j){g) < 0}. We observe that the hypothesis that go implies 
that V^4'{gQ) 7^ 0, and therefore, by possibly restricting U we can assume that V^0(5) / for 
every g £ S CiU. We thus have 

(9.17) N^{g) = V"(l){g) , for every g £ S nU , 
and therefore 

(9.18) = for every g £ S nU . 



Proposition 9.12. At every point ofSnU one has 

\VH^fn = {\Vh^\^ Ah(I> - Ah,oo<P} ■ 

Proof. We use the summation convention over repeated indices. Invoking Proposition 19.91 and 
1)9. 18(1 . we obtain at every point in 5 \ S 



H(, 



Ah, 



□ 



It is interesting to consider a nonlinear operator which interpolates in an appropriate sense 
between the iif-mean curvature operator in Definition 19.81 and the operator Ah^oo- Consider 
the one-parameter family of quasilinear operators defined by 

(9.19) Ah,pU = divH{\V"u\P-^V"u) =0, 1< p < oo . 
Supposing that |V^u| 7^ 0, we formally rewrite in the more suggestive fashion 

(9.20) Ah,pU = (p-2) \V^ur^ \V^u\^ Ahu + Ah,ooU^ . 
If Up is a solution to An^pUp = 0, then equation (|9.2flj) gives 

iV-^Upp AnUp + An^ooUp = . 
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If we assume that Up — > u^o as p — > cxd, and that |V^UppA/^Up is bounded independently of 
p large, then by letting p — > oo we formally find that Uoo must be a solution to Aj^^oo^oo = 0. 
On the other hand, if we know instead that Up ^ ui as p ^ 1, then we discover from H9.2U() and 
Proposition 19.121 that 

^H.pUp — > - n{ui) , 

where Ti.{ui) is the H-mean curvature of the level sets of uil This suggests that one should 
study the behavior as p — > 1 of the one-parameter family of quasilinear operators ^H,p- 

Comparison with S. Pauls' notion of horizontal mean curvature. In the first Heisenberg 
group another notion of horizontal mean curvature was introduced by S. Pauls in Pa_. Such 
notion is based on the procedure of Riemannian e-regularization defined in the proof of Theorem 
18.51 Using 1)8. 5|) one sees that, given a function (j) G C^(EI"), its gradient with respect to the 
metric (gfj) is given by 

(9.21) V,(P = XkP Xi + X2(p X2 + T,(j) T, . 
Let us note in passing that ()9.21|) gives 

(9.22) |V,(/.p = \Vh(P\'^ + €{T(j)f , and = Ah(P + eT'^cj), 

where we have denoted by A^ the Laplace-Beltrami operator with respect to the metric (glj). 
In |Paj the author defined the horizontal mean curvature of 5 at a point go & S\T,s as follows 



(9.23) Hpigo) lim , 

where indicates the mean curvature of S in the Riemannian metric 1)8. 4() . We now recognize 
that such notion coincides with the one introduced in Definition 19.81 

Proposition 9.13. The horizontal mean curvature defined by (|9.23j) coincides with the one in 
Definition \y.tA 

Proof. Let 5 C be a surface. The Riemannian Gauss map of S with respect to the 
metric is given by u'^ — \N'^\ ' ^^^^^ we have denoted by |iV''|e the length of in such 
metric. Let us notice that |A?"'^|^ = ■^^/W^^^^\^l'uP . Recalling (|8.7jl we see that 

(9.24) u, = a' {pXi+qX2 + ^feujT,] , 



where we have let = Wj^W^ ^eu?. From (Pni) and (tTX^ below, we recognize that the 
expression of the Gauss map in the Cartesian coordinates (x, y, t) is given by 

/ _ _ _ _ _ 

IV, = ( a' p , a' g , e q' w + —{xq - yp) 

Using Proposition EHl (!5.1H|) . and the fact that det{gfj) = e^^, we then see that at any go ^ S 

a' 

= Xi{a' p) + X2{a' q) + e T{a' ZJ) 

= a^n + pXi{a') + qX2{a') + e{a'TLJ + oJT{a')) , 

where we have used the fact that 7i = Xip + X2q, see Definition 19.81 and Proposition 19.91 We 
now claim that at any go S 5 \ S we have a*^ ^ 1 as e — > 0, and that furthermore the following 
cancelation relations hold 

(9.25) Xi{a') 0, Xi{a') 0, T{a') ^0, as e^O. 



Hniao) = diVeV^ = d^{a^ p) + dy{a^ q) + dt{e u + —{xq - yp)) 
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We only prove the first relation of ()9.25|) , leaving the analogous details of the remaining two 
to the reader. We have 

, _ {W^ + euj'^)XiW - WjWXiW + eioXiUj) 

~ + 6^2)3/2 

ujXiW - WXiLO 

From (|9.25j) we conclude that TCp{go) = hni H^{gQ) = ^{{go), at every go G 5 \ S. 

□ 

Comparison with the notion of pseudo-hermitian mean curvature of Cheng, Hwang, 
Malchiodi and Yang. In jCHMYj the authors have introduced the following notion of pseudo- 
hermitian curvature for a surface S C M, where (M, J, 0) is a three-dimensional oriented CR 
manifold, with CR structure J, and global contact form Q. At every point g £ S \ S5, they 
consider the one-dimensional space HTgS. They fix a unit vector field ei G HTS with respect to 
the metric G = ^d0(-, J ) associated with the Levi form, and then define 62 = J(ei). They call 
62 the Legendrian normal or Gauss map. They denote by V^ ^*' the pseudo-hermitian connection 
associated with J, 0). There exists a function H^-^- such that 

(9.26) V^fei = H^-^- 63 . 

Such function Hp ^ is called the pseudo-hermitian mean curvature of S, see (2.1) in |CHMYj . 

Proposition 9.14. Let M he the Heisenberg group , then the function H^-^- coincides (up to 
a choice of the orientation) with the horizontal H-mean curvature in Definition \y.8l 

Proof. One can check that in the Heisenberg group the pseudo-hermitian connection V^''^' is 
nothing but the horizontal Levi-Civita connection introduced in section |^ Since a basis 
for HTS is given by ei = (u^)-^, see Corollarv 19.51 and we clearly have 62 = , from the 
horizontal Koszul identity ()5.14|) we obtain for every vector field X = aei + be2 + cT 

(9.27) 2<V^^ei,X> = 2ei < ei, X > -X < ei, ei > 

- 2<eu[euXf > + <X,[ei,eif > 
= 2ei(a) - 2 < ei,[ei,Xf > . 

We now have 

[ei,X] = ei{a)ei + ei{b)e2 + b[ei,e2]+ei{c)T + c[ei,T] . 

The commutators in the right-hand side of the latter equation have been computed in section 
^Jbelow, where the vector fields ei, and 62 are respectively denoted by Z and Y. From Lemma 
113.81 and Lemma 113.91 we find 

[ei,X] = ei{a)ei+ei{b)e2 + b\T + nei + {qe2ip)-pe2iq))e2\ +ei{c)T + c{qTp-pTq)e2 . 



From the latter expression we obtain 

[ei, X]" = ei(a)ei + ei{b)e2 + bi^Hei + {qe2{p) - pe2{q))e2^ + c{qTp - pTq)e2 



and therefore (|9.27|) gives 



< V^ei,X > = ei(a) - ei(a) - bU = < -'He2,X > 



ei 

From the arbitrariness of X we conclude 
(9.28) V^^ei = - W 62 . 
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This proves the proposition. 

□ 



10. Sub-Riemannian calculus on hypersurfaces 

In this section we estabUsh some basic integration by parts formulas involving the tangential 
horizontal gradient on a hypersurface, and the horizontal mean curvature of the latter. Such 
formulas are reminiscent of the classical one, and in fact they encompass the latter. However, 
an important difference is that the ordinary volume form on the hypersurface S is replaced by 
the if-perimeter measure dan- Furthermore, they contain additional terms which are due to 
the non-trivial commutation relations, which is reflected in the lack of torsion freeness of the 
horizontal connection on S. Such term prevents the corresponding horizontal Laplace-Beltrami 
operator from being formally self-adjoint in L'^{S,daH) in general. Since the framework we 
work in does not lend itself to a preferred choice of coordinates, for ease of computation we have 
developed an approach, based on Federer's co-area formula, which is coordinate- free. In fact, 
our proof slightly simplifies several of the classical formulas for hypersurfaces in which are 
derived by writing 5 as a graph, see e.g. |MMj . |Gij . 

Theorem 10.1 (First sub-Riemannian integration by parts formula). Consider a 
oriented hypersurface in a Carnot group S C G. If u £ Cq{S \ S^), then we have 

(10.1) J Vf'^u dan ~ J ^ ~ ^^'^1^'^^^ > ^ = l,...,m , 

where the functions c^'^ on S \ Tj are defined by 

k m 

(10-2) cf'^ = EiE^hp^P^^ 

s=l j=l 

with bfj denoting the horizontal group constants defined in (|2.14() . Moreover, the horizontal 
vector field c^'^ = '}2T=i ^f'^-^i ^-^ perpendicular to the horizontal Gauss map , i.e., one has 

(10.3) < c^'^,iy^ > = . 

As a consequence, we have c^'^ G C^{S \ Tig, HTS). 

Proof. Since the question is local, to prove the theorem we will assume, without loss of gener- 
ality, that S is the level set of a defining function (j), and that S is oriented in such a way 
that AT" = V(/>. Furthermore, thanks to the assumption of the support of u, we can also assume 
that S be non-characteristic. Using a partition of unity we can always reduce ourselves to this 
situation. For every p E M, we define Up = {g £ G \ 4)[g) < p}. By the non-characteristic 
assumption on S we can assume that, if 5 = dUp^, then for every p sufficiently close to po the 
characteristic locus of dUp is empty. Federer's coarea formula gives, see [F^ . 

(10.4) [ Vf'^u W dg = r [ V^'^u dHN-i dr = T f vf"^u dan dp . 

JUp J-ooJdUr \^\ J-ooJdUr 
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Recalling and I^TT)) . we obtain from ((THH) 

(10.5) / Vf "^n duH = [ Vf"^u W dg . 

JdUp dp Jii^ 

This crucial observation allows us to reduce the computation of the surface integral to that 
of an integral over the solid region Up. Recalling Definition 17.41 we have 

(10.6) [ Vf "^u W dg = [ XiuW dg - [ Xju pj W dg , 

JUp JUp JUp 

where we have adopted the summation convention over repeated indices. Integrating by parts 
in the first integral in the right-hand side of H10.6() we find 

(10.7) / XiuWdg= [ u< N,Xi> ^ dHN-i - f u XiW dg 
JUp JdUp \^\ JUp 

upiW dan - I u ^^^^ W dg , 

dUp JUp ^ 

where we have used div Xi = 0, see 1)5. 16|) . We next integrate by parts in the second integral in 
the right-hand side of (|lfl.6|) . obtaining as in (|10.7|) 

(10.8) / XjU Pj Pi W dg = / u Pj pj Pi W dan 

JUp JdUp 

u Xj{pj Pi W) dg 

Up 



/ upiW dan - u (XjpA pi W dg 

JdUn JUn 



Inserting ()1U.7|) . (|1U.8|) into (|1U.6I) . we see that the boundary integrals disappear and we finally 
obtain 

(10.9) / vf'^uW dg = [ u Xj{pj)piW dg - [ u cf W dg , 

JUp JUp JUp 

where we have let _ 

cf = ^{Xip,-X,p,] . 

Formula ()1U.9|) is the crucial point in the proof. Proceeding now as in (|10.4() . and applying 
(|10.5j) . we conclude for every /? € M in a sufficiently small neighborhood of a given /jq G K 

(10.10) / vf'^u dan = u XjipA Pi dan — u cf dan ■ 
JaUn JaUn JdUp 



We now have 



us 



s=i \i=i 

Recalling ()9.11|) in ProDosition l9.91 we conclude that H10.1() holds. Finally, H10.3() follows from 
the skew-symmetry of the matrix {6|j}ij=i,...,m defined by (|2.14|) which gives XlJLi MjPiPj — 
for every s = 1, k. Hence, 

k m 

< c^"5,i.^ > = E {Y.^tjmPs = . 

s=l j=l 
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This completes the proof. 

□ 



Remark 10.2. We emphasize that in the Abelian case G = W^, we have Xi = d/dxi, i = 
l,...,m, and so [Xi,Xj\ = and thereby cf = 0. In this case formula ()1U.15() recaptures the 
classical integration by parts formula on a hypersurface, see for instance .MM] , pij . 

Remark 10.3. We note explicitly that when G = H", the Heisenberg group, then the horizontal 
vector field c^''^ is given by 

(10.11) c^"5 = lJ J{u^) , 

where J : HIF^ — > HW^ is the symplectic transformation which, in the orthonormal basis 
{Xi, X2n} ofHIF^, is represented by the block matrix 



-I 0, 



J 



We thus obtain from (jlU.llf) 

(10.12) C^"5 = uJ (U^)^ = ZJ {pn+lXl + ... +P2nXn-PlXn+l " ... -p„X2„) . 

Therefore, for H" formula ()1U.1|) reads 

(10.13) j V^'-^u dan = j u - uJ {u^)^^daH ■ 

In particular, when n = 1 then in the notation of section see also Remark \6.'A we have 
c^'S _ jjj2; ^ and we can write (|l().13j) as follows 

(10.14) / V^"^u dau = [ u InY - uJ zXdan ■ 



Is Js 
We have the following notable consequences of Theorem llU.ll 

Theorem 10.4. Let S d G be a C"^ oriented hypersurface, with characteristic set S. If C ^ 
Cq{S \ Tis, HTS), then we have 

(10.15) / {divH,sC + <c^,C>} dan = [h <C,t^^> dan, 
Js Js 

where we have let 

m 

divH,sC = ^"^i^'^Ci ■ 
1=1 

Theorem 10.5. In a Carnot group G suppose that the hypersurface S is a vertical cylinder as 
in Provosition \y.(A If u £ Cq(5) we have 

(10.16) [ Vf'^u dan = [ uH dau ■ 

Js Js 

Proof. First of all we notice that the assumption on S guarantees that the characteristic set 
is empty, see Proposition 19.61 It is thereby legitimate to assume u £ Cq{S), instead of 
u G Cg(5 \ S5). Next, we observe that since the defining function of S depends only on the 
horizontal variables, then the normal has no component along V2, and therefore uJs = for 
s = 1, k. This implies c^'*^ = 0, see p0.2() . The conclusion thus follows from H10.15() . 

□ 
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We next establish another integration by parts formula which involves differentiation along 
a special combination of the vector fields and T^, s = l,...,k, where Tg constitute the 
orthonormal basis of the first vertical layer defined in H2.16() . Such result plays a central role in 
the last two sections of this paper. 

Theorem 10.6 (Second sub-Riemannian integration by parts formula). Let S be a 

oriented hypersurface in a Carnot group G. For every f,( G Co(ZY \ T,s), where U C G is an 
open neighborhood of S, then one has for s = 1, k 

(10.17) / / {TsC-iOs < >)daH = - [ C {Tsf-iOs < V/, >) dan 
Js Js 

f ( m-i m "j 

+ J^f C^uJsn + I^I^fe^3/| dau , 

where Us are defined in (|6..S|) . and for i = l,...,m, s = l,...,k and i = l,...,m3, we have let 
b^g =< [ei,es],e3^£ >. In particular, for a hypersurface S C H", we have k = 1, and therefore 

letting uJi =U and setting Yf < V/, >, see also section UM we obtain, 

(10.18) / / (r-cjy)C dan = - [ C {T-uJY)f dan + / fQUHdan • 
Js Js Js 

Proof. We use the same idea of the proof of Theorem llO.il except that this time we consider 

/ [TJ- <V{u:sf),v'' >\Wdg = f <Ts,N> f^ da - [ f div{WTs) dg 
J Up JdUp M^l J Up 

-I <u^,N > cj,/-^ da + [ uJsf div{Wu") dg 
JaUp Jup 



lav 



UJsf dan - I f Wdg 



[ WuJsf dan + [ uJsfW div{u") dg + [ cJ,/ < ^^^""^ ^ Wdg , 
JaUp J Up Jun ^ 

e have used the identity < 
terms drop and we are left with 



laUp J Up J Up 

where we have used the identity < , N >= W, see 1)6. 10() . Since ZJsW = oos, the two boundary 



r r T W 

/ [Tsf- < V(uJ,/),i/^ >]Wdg = - f ^Wdg 

+ UsfW div{v") dg + / Usf Wdg . 

JUp JUp ^ 

Using the coarea formula as in the proof of Theorem llU.ll and differentiating the resulting 
integrals, we obtain from the latter identity 



TsW 
W 



dan 



(10.19) / [Tsf-<V{ujsf),v'' >]daH = - [ f 

Js Js 

+ J UJsf div[u ) dan + J t^sf dau 

Since Definition 19.81 and Proposition 19.91 give 

m 

n = divH,s{v^) = Y^^^Pi = dtv{i^") , 
1=1 
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we can re-write ()1U.19() as follows 

(10.20) / [Tsf-ujs < V/,i/^ >]daH = f f < VcJ„i/^ > dan 
Js Js 

+ j^u^JUdau - jj [-^-^s ^ J daH. 

We now observe that ()6.3p gives 

(10.21) < VuJ^i.^ > = ^ uJ, ^ . 

On the other hand, we have 

(10.22) = + 2^2^h,p,u:^^,. 

e=i i=i 

To prove (|10.22j) . suppose, as we may, that S is locally described as the zero set of a 
function 0, and that N = = Yl^iPi-^i + Z]s=i '^sTg + Yl^j=3 YlT=i ^iZ-^j/- i^^is have 

m 

1=1 

m m 

= Y.p,Ts{X,(p) + W ^Y.^lp,oJs,e 

i=l 1=1 1=1 

m mg m 

= Y.p,Ts{p,W) + W Y,Y.^Ip,^^,, 

1=1 1=1 i=l 

(m \ / m \ m3 m 

i=l / \i=l J 1=1 1=1 

ma m 

e=i i=i 

where we have used the commutation relations [Xj,T<j] = ^^x^L"^3>^' This proves (|10.22() . 
Inserting (|10.22jl in (|10.21|1 . and the resulting equation in (|10.20j) . we reach the conclusion 

/ {TJ -oJs< V/, u"" >)daH = fTvsndcjH + f ^IPi^^/ dan ■ 

If we replace / by fC, in the latter integral identity we obtain the sought for integration by 
parts formula. 

□ 



11. Tangential horizontal Laplacian 



In this section we introduce a tangential partial differential operator, IS.h,s (and a modified 
version of the latter) , which constitutes the sub-Riemannian counterpart of the classical Laplace- 
Beltrami operator on a hypersuface. In fact, as we will see, it reduces to the latter when the group 
G is Abelian. It has however one aspect which distinguishes it from its classical predecessor, 
and this is lack of self-adjointness in L^{S, dan)- This phenomenon is caused by the presence of 
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the "drift" term c^''^ in the integration by parts formula in Theorem llU.il In the next section 
we win show that the horizontal mean curvature flow recently proposed by Bonk and Capogna 
|BCj satisfies a nonlinear pde which involves the operator Aj^g, see Theorem 112. II 

Definition 11.1. Given a function u £ C'^{S), the tangential horizontal Laplacian of u on S 

is defined as follows at points of S \ T,s 



(11.1) /^H,SU = l^^i' ^i' 



u . 

L t, 

i=l 



We also introduce the modified tangential horizontal Laplacian on S 

(11.2) Ah,su =^ Ah,su + < c'5,V^"5n > , 
where c^'^ is given by H1U.2|) . 

Remark 11.2. One should keep in mind that when S is a vertical cylinder given by ()9.9|) . then 
the operators Ah^s and Ah^s coincide 

Ah,s = Ah,s ■ 

In such case it is easy to show from Theorem MU. 1\ that Ah^s is formally self-adjoint in L^{S, dan)- 

One basic raison d'etre for the operator Ah^s is in the following sub-Riemannian Stokes' 
theorem which follows from Theorem llO.il 

Corollary 11.3. Let u G Cg(5 \ S5), then we have 

(11.3) ^ Ah,su dan = . 

Proof. It suffices to take V • ' u instead of u in Theorem IIU.II and then add the resulting 
identities in i = 1, ...,m. Keeping in mind the definition ()11.2|) . formula 1)10. 15|) gives, 

/ Ah,su dan = n< V^"^u, > dan = , 
Js ' Js 

since by (|9.6|) one has < V^''^u, >= on 5 \ S5. 

□ 

Corollary 11.4. Let u G C^{S), then for every ( G Cq{S \ S5) we have 

(11.4) / <V^'^u,V^'^C > dan = - [ u Ah,sC dan ■ 
Js Js 

Proof. We take u ' C> instead of u, in Theorem llO.il 



□ 



Remark 11.5. In connection with Remark \y.ll\ we see that for the product group G = G x 
one has Ah^s = Ah^s = Ah on S = G x {0}. 



The following formulas are verified by direct computation from the definition. 
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Lemma 11.6. Let u,v £ C^iO), F £ C^{R), then we have on O \ S 

(11.5) AH,siuv) = u Ah,sv + V Ah,su + 2 < V^'^u,\/^'^v > , 

(11.6) AH,siFou) = (F"oti) |V^"^n|2 + {F' ou) Ah,su . 

The next result provides a useful mean for computing the operators Ah,s and Ah,s on S, 
using the vector fields Xi, ...,Xm in the ambient group G. 

Proposition 11.7. Let u G C^(5), then we have on S \ T, 

(11.7) Ah,su = Ahu - <VjjU > - <V"u,u" > Ti, 

(11.8) Ah,su = Ahu+ <c"'^,V"u> - <Vjju > - <V"u,u" > Ti , 

where u denotes any extension of u. In the above formulas, the notation VjjU indicates the 
horizontal Hessian ofu introduced in ()9.15l) . 

Proof. We begin with Definition 17.41 which gives 

where u is any extension of u. Applying (|11.H) . and using the summation convention over 
repeated indices, we find 

(11.9) Vf'^Vf'^^x = Vf'^(X,n) - Vf'^(< V''u,u^ >) - < V''u,u^ > vf'^ z.f . 
We now compute the terms in the right-hand side of (|11.9)) . 

(11.10) VP(Xin) = XiXiU - < V^{Xiu),u^ > i/f 

= Ahu - XjXiU vf 

= Ahu - <Vhu > . 

Next, equation (|6.7j) gives 

(11.11) Vf"^(< V^u,iy^ >) i/f = Xi{XjU vf) - < V^(< V^u,u^ >),u^ > i/f z^f 

= . 

Finally, we find from Proposition 19.91 

(11.12) < v^M, > vf = < V^u, > n . 

We now substitute (|11.1U() , (jll.llf) , (|11.12() in ()11.9|1 . To reach the desired conclusion we only 
need to observe that thanks to (|1U.2|) one has 

< c^"5, V-^"^u > = < c^'^,V"u > - < V^u,iy" >< c^'^,u" > = < c"^^,V"u > . 

□ 

The first elementary example of solutions of the tangential operators Ah.s and Ah^s is pro- 
vided by the following consequence of Proposition 111.71 



Proposition 11.8. // the function u is constant on S, then 

^H,su = Ah,su = . 
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Proof. First of all, let us notice that, since A}{,su and Ajj^su only depend on the values of u 
on S, we can without restriction assume that u = 1 in G. Under such hypothesis the conclusion 
now follows trivially from Proposition 111.71 

□ 

Another interesting consequence of Proposition II 1 . 7l and of the grading structure of a Carnot 
group is the following. 

Proposition 11.9. Let S C G be a H-minimal hypersurface, then if x{g) = {xi{g), ...,Xmig)) 
denote the projection onto the horizontal layer of the exponential coordinates of g ^ G (see 
(|2.12j) ), one has 

'^H,sixi) = , i = l, ■■■,m . 

Proof. From Proposition l5.7l we have Anixi) = 0, and also V|^(xj) = 0. The desired conclusion 
thus follows immediately from pi.7|l . 

□ 

We next analyze a situation of special interest, namely when G is a Carnot group of step 
r = 2, and one has a hypersurface S given as a graph over the first layer of the Lie algebra. In 
such case, identifying via the exponential map g = exp ^{g) with ^(g) = {x{g),t{g)), we can 
find an open set J7 C Vi, and a function /i : $7 — > M, such that for some s G {1,...,A;}, S can 
be written as 

(11.13) S = {ix{g),tig)) £ G \ x{g) G , ts{g) = h{{x{g))} . 

For instance, in the special case of the Heisenberg group H"" we would be considering a graph 
over R2", i.e., S = {{x,y,t) G | (z, y) G J7 C M^" , t = h{x,y)}. 

Theorem 11.10. Let G be a Carnot group of step r = 2, and S d G he a H-minimal hy- 
persurface of the type (|11.13p . then outside the characteristic set Tig the coordinate functions 
xi, Xm, ti, tfc are solutions of the tangential sub-Laplacian onS. 

Proof. For the horizontal coordinates xi,...,Xm the conclusion follows from Proposition 111.91 
We now recall ()5.23() in Proposition 15.71 

(11.14) Xits = ^ < [^i,ej],es > , Auts = 0, i = 1, m , s = 1, A; . 
The first equation in ()11.14|) can be written 

^ m 

i=i 

Thanks to (|5.21l) this gives 

XjXits = ^ < [ej,ei],es > = ~ ^ < [ei,ej],es > = - XiXjts , 
and therefore for every s = 1, ...,k, one has 

(11.15) Vlits) = . 

Now Proposition 111.71 gives for any / G {1, A;}, with I ^ s 

AhMi = ^Hti- <vjj{ti) u^,u" > - <v^h,u"> n = 0, 

when S is ff- minimal, thanks to (|11.14j) and (|11.15jl . We are left with proving that, if S is 
//-minimal then An^sits) = 0. Since on S we have tg = h{x), we need to show that An^sh = 
on S. With this objective in mind we begin by expressing the iZ-mean curvature of S in terms of 
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the function h. We consider the function (/>(g) = tg — h[x) defining S. According to Proposition 



19.121 one has on 5 \ , 

(11-16) H= ^i^llV^cAl'A^^,/. - A^^,oo0} . 

On the other hand, we have from Proposition 111.71 
(11.17) l^H,sh = Anh - KVj^h > - <V^h,u^ > H 

^ {|V^0PAh/i- < V^/i V^(/),V-^0 >} - <V^h,u^ > n 



|V^0| 



2 



^ { I V^0|2 Ah(/i - t,) + Ants - <Vlih- ts) V"cj), > 



|V^(/.|2 

- < Vj^its) V">p,V"(t) > } - < V"h,u^ > n . 
If in (|11.17() we use (jll.lSp and the second equation in 1)11.14^ . we obtain 

(11.18) AH,sh = - ^^{|V^0|2Aj^(/> - Ah,oo^} - <V^/i,r/^> H. 

We now compare (|11.18j) with (|11.16j) to reach the following interesting conclusion 

(11.19) AH,sh = - {|V^</.| + < V^h,iyH >} n . 

It is now clear from (|11.19|) that if H = 0, then Afj^sh = 0, and this completes the proof. 



□ 



Corollary 11.11. In the Heisenberg group let 

S = {{x,y,t)eir\ {x,y)£n , t = h{x,y)} , 

where Q C M^" is an open set, and h G (7^(17). Is S is H -minimal, then the coordinate functions 
xi, Xn,yi, ■■■,yn,'t are solutions of Ah,s onS. 

Corollary 11.12. Let G he a Carnot group, and consider the exponential horizontal coordinates 
xi{g), ...,Xm{g) in G, then 

AH,s{xi) = - <u^,Xi> H = -piTi, i = l,...,m. 

Consider the exponential coordinates ti{g), ...,tk{g) in the first vertical layer V2, then 



^ m 

AH,s{ts) = - 2 ^ ^« ^ ' ^ = 1' •••'^ • 

In particular, when G = 'M^ , then 



AH,s{t) = - ^ (xp + yq) . 



We close this section with introducing the notions of p-Dirichlet integral and of p-harmonic 
function on an hypersurface. Such notions play a central role in the development of geometric 
sub elliptic pde's on hyper surf aces in Carnot groups. 
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Definition 11.13. Suppose that S C G be a C'^ hyper surf ace, with = 0. Given 1 < p < oo 
we define the p-Dirichlet integral of a function u G Cq{S) as 

£h,sN = - f |V^"5^xr dan . 
P Js 

Suppose that u S L^{S,daH), and that moreover Vj ' u G L^{S,daH), for i = \, ...,m. We say 
that u is p-subharmonic ('-superharmonic j in S if for every ^ G Cq{S), C ^ 0, one has 

J |V^"5n|P-2 <V^'^u,V^'^(:> dan < (> 0) . 

We say that u is p-harmonic in S if u is simultaneously p-suhharmonic and p-superharmonic. 
When p = 2 we simply say that u is subharmonic, superharmonic or harmonic in S. 

According to Corollary 111.41 we can adopt the following alternative notion of subharmonicity. 
Definition 11.14. A function u G Lj^^{S,dafj) is called subharmonic in S if 
(11.20) ° - ^"'^^ '^''^ ' ^""^"^^ ^ ^ '^^^^^ ' C > . 



12. Flow by horizontal mean curvature 

In connection with Proposition II 1 . 7l we recall the Riemannian counterpart of H11.7() 

(12.1) Amw = An - <V'^uv,u> - {n - 1) < V mu, u > H, 

where Am and V m respectively represent the Laplace-Beltrami operator and the intrinsic gra- 
dient on an (n — 1) -dimensional Riemannian manifold M. Formula 1)12.1(1 plays a crucial role, 
for instance, in the derivation of the equation for flow by mean curvature, see for instance [E]. 
If one considers a family of smooth embeddings F{-,t) : M — > M", then with = F{M, t), the 
equation of flow my mean curvature is given by 

dF 

(12.2) —{p,t) = -{n-l)Hu . 

If we write x = F{p, t), then using (|12.1|) and ((12. 2() we obtain the nonlinear partial differential 
equation 

dx 

(12.3) = Am,x , 

which is satisfied by the components (xi, ...,x„) of a;. This can be readily recognized as follows. 
Equation ()12.1|) gives for each component Xi 

AuAxi) = A{xi) - <V^{xi) u,iy > - (n - 1) <V{xi),u> H 
= - (n - 1) <ei,u> H = -{n-l)uiH . 

In other words, we have A^tX = — (n — l)Hv. This equation, combined with (|12.2|) . proves 
(fTTTl) . 

We next want to prove a sub-Riemannian analogue of (|12.3|) for the mean curvature flow in 
the Heisenberg group recently proposed by Bonk and Capogna in |BCj . We consider a smooth 
hypersurface in a Carnot group S C G, and a family of smooth embeddings F : 5 x (0, T) — > G. 
We will denote by 5'*' = F{S, A). The reader should note that we are using the unconventional 
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parameter A G (0, T) to indicate time. The reason is due to the fact that, to keep a homogeneous 
notation with the Heisenberg group, we have already reserved the letter t = (ti, t^) to indicate 
the exponential coordinates in the first vertical layer V2 of the Lie algebra of G, see ()2.12() . In 
|BCj the authors have introduced the following definition of horizontal mean curvature flow 
when the group G is H". At any point F{g, X) ^ \ T,^ (S^ denotes the characteristic set of 
S^), they require that 

OF 

(12.4) <—^N>= -n <i^",N> . 

OA 

We notice that it is important to project the flow along the normal direction since the vector 
equation ^ = —Tiv^ is meaningless: the right-hand side evolves in the horizontal bundle HG, 
whereas the left-hand side has components which move outside of it. Also, as noted in pQ. "any 
tangential component of the velocity field only gives rise to a re-parametrization of the surface 
with no effect on the geometric evolution". At characteristic points the equation ()12.4|) is not 
defined and the way the authors circumvent this obstacle is by restricting to S the Riemannian 
e-regularization of the sub- Riemannian metric of S introduced in (|8.4|) . We refer the reader to 
|BCj for the relevant details. We want to next prove the following result which underscores the 
interest of the operator /S.h,s introduced in the previous section. It should be thought of as the 
sub-Riemannian analogue of (|12.3|1 . 

Theorem 12.1. Let F : S x {0,T) ^ G be a solution of the horizontal mean curvature flow 
(|12.4() . then at any non- characteristic point F{g, A) G S'^ one has 

OF 

(12.5) < AT > = < Ah^sxF, N > , 
where the latter equation must be interpreted component-wise. 

Proof. To make our proof as transparent as possible we discuss in detail the case of the first 
Heisenberg group H^. The details of the more general case, as well as some applications of 
1)12. 5() . will appear elsewhere. We consider F{g, A) = {x{g, A), y{g, X),t{g, A)) and notice that we 
have from (|13.2|) below, 

(12.6) Ah,s^F = iAH,s^{x),AHMy)^^HMt)) 

= Ah^sx{x)Xi + AHsx{y)X2 + I A^_5A(t) + 1 T 

At this point we use 1)12. 6(1 and the fact that 



< N T > 

N = [pXi + qX2 + T] W , 



to discover that 



(12.7) < A^,5aF, AT > = W S^p Ah^s,{x) + q Ajj^s^iy) 



<N,T> f yA^5A(x)-xA 

(y) 

+ 777 [^HMt) + 



We now use Corollarv lll.121 which in the present situation gives, 
(12.8) Ah,s^{x) = -pH, AH^s^iy) = -qU, AH,s,{t) = - ^S^J^ n . 
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Substituting 1)12. 8p in 1)12. 7(1 we obtain the remarkable conclusion 



(12.9) <Ajj^sxF,N> = w!^-fn-fn 

< N,T > f xq- yp xq - yp 
+ w [ ^^ + ^^^ 

= -wn . 

On the other hand, gives 

<u^,N > n = wn . 

Combining the latter equation with 1)12. 9() we reach the conclusion 

(12.10) <A^5AF,Ar>= -n <u^,N> . 
Finally, from (|12.1UI) and ((TTl)) we obtain ((n3)) 



□ 



13. Some geometric identities in the Heisenberg group 

In this section we collect several geometric identities in the Heisenberg group which, 
besides their intrinsic interest, play an important role in the development of the first and second 
variation formulas in Section [TH We note preliminarily that 

(13.1) Xl^X2 = T, X2AT = Xi, XiAT = -X2, 

where the wedge products are computed with respect to the left-invariant Riemannian metric 
with respect to which {Xi, X2,T} constitute an orthonormal basis. We also observe that the 
passage from the orthonormal basis {Xi, X2,T} to the standard rectangular coordinates of 
is given by the formula 

(13.2) aXi + bX2 + cT = (^a,b,c + 

Throughout this section 5 C denotes an oriented surface, with non-unit normal N, 
and Riemannian Gauss map v, we consider the functions pi,p2 and W on S defined in (|(i.lj) . 
As we have mentioned in Remark 16.21 for computational ease it will be convenient to adopt in 
this and the next two sections the slightly different notation p = pi,q = P2, i-e.. 



(13.3) p=<N,Xi>, q=<N,X2>, W = Vp^+q'^ ■ 
The horizontal Gauss map defined in 1)6. 6|) is now given on 5 \ S by 

(13.4) = pXi + qX2 , 
where we have let 

(13.5) p = ^ , q = A. ^ so that + q^ = 1 on S ■ 

We also introduce the notation 

(13.6) Lo = <N,T> zj=-^. 

W 
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We notice explicitly that if 5 G , k > 2, then p,q,u>,p,q,io G C^^^{S \ S^). We also note 
that, thanks to Proposition I9.9[ the if-mean curvature of S is presently given by the formula 

(13.7) n = Xip + X2q . 

Along with the horizontal Gauss map we consider the vector field 

(13.8) {u")^ = = qX^-pX2, 

which, as already noticed in Sectional constitutes a basis of HTS. It will be convenient to keep 
a different notation for the action of the vector fields , (t^^)'^ on a function ( £ Cq{S \ S^). 
We thus set 

(13.9) YC < VC, > = P XiC + qX2C, 



(13.10) ZC =^ < VC, {u^)^ > = q XiC - P X2C . 

We mention that in the right-hand sides of (|13.9|) . (|13.1U|) the vector fields Xi, X2 act on 
an extension C, of C- However, for the sake of simplifying the notation we have used, and will 
continue to do so below, the same notation for both functions. It is worth observing that 
{Z, Y, T} constitutes an orthonormal frame on S. One has in fact 

(13.11) Z AY = T , YAT = Z, TAZ = Y. 
Moreover, the (Riemannian) divergence in M.^ of these vector fields is given by 

(13.12) div Y = Xip + X2q = H , div Z = Xiq - X2P . 
Using Cramer's rule one easily obtains from (|13.9j) and p3.1()|) 

(13.13) Xi = pY + q Z , X2 = qY - p Z . 
One also has 

(13.14) Vf"^ = q Z , = -p Z . 
To prove (|13.14|) we proceed as follows 

Vf'-^C = XiC- < V^C,'-'' >i^f = XiC - {pXiC +qX2C)p 
= XiC - f XiC -pq X2C = t XiC - pq X2C = q {q XiC - p X2C) 
= qZC, 

Vf'-^C = X2C- KV^Cu'^yu^ = X2C - {pXiC +qX2C)q 
= X2C - pq XiC - t X2C = p^ X2C - pqX.C = - P {q XiC - P X2C) 

= -pzc. 

These formulas give 

(13.15) V^'^C = q ZQ Xi - p ZQ X2 . 
From ()13.15() and ()13.5|) we obtain 

(13.16) IV^'-^CI' = {ZCf = {qXiC - pX2(f . 

We next establish some identities that will be used times and again in Sections 1141 and [T31 
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Lemma 13.1. One has on S \ 

(13.17) pZp+ q Zq = p Yp + q Yq = p Tp + q Tq = 0, 

(13.18) p Z^p + q Z\ = - [Zpf - {Zqf . 

It is useful to note the following alternative expression of the first two identities in 1)13. 17|1 

(13.19) pqXip- p'^X2P + q'^Xiq -pqX2q = , 

(13.20) f Xip + pq X2P + pq X^q + f X2q = . 

Proof. The proof of 1)13. 17|) follows trivially by differentiating the identity -l-'tf = 1, whereas 
1)13. 18(1 follows by differentiating pZp + qZq = with respect to Z. One has from (|13.17(1 and 

(|mn|) 

= p Zp + q Zq = pi-q Xip - p X2P) + q{q Xiq - p X2q) , 
which proves (|13.19|) . Similarly, 

= pYp + qYq = p{p Xyp + q X2P) + q{p Xiq + q X2q) , 
which implies (|13.20j) . 

□ 

Lemma 13.2. One has on S\Tis 

(13.21) <Z,Ar>=0, <Y,N>=W, 

(13.22) Yu = TW , 

(13.23) qYp - pYq = X2P - Xiq , 

ZW 

(13.24) = qYp - pYq + uj , 

and 

(13.25) = qTp - pTq . 

Proof. The first identity in 1)13. 21|) is obvious, while the second one is simply a reformulation 
of (|6.1U|) . The identity (|13.22() is just a special case of (|1U.22() . To prove (|13.23|) . it suffices to 
use (jmi and (tm?!) to find 

X2P — Xiq = qYp — pYq — {pYp + qYq) = qYp — pYq . 

As for (|13.24l) we have 

uj = T()) = XiX2(t) - XiXic/) = Xi{qW) - X2{pW) = - {X2P - Xiq) W + ZW , 

from which the desired conclusion follows immediately. 

Finally, we turn to the proof of (|13.25() . Applying T to both sides of (|13.21() we obtain 



= T{Z^) = T{qXi^ - pX2^) = TqXicj) + qTXi^ - TpX2^ - pTX2^ 
= TqXi(t)-TpX2(l) + qXiT(t)-pX2T(t) = pTq - qTp + Z{Tct>) . 
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It follows that 

Z{T<P) 

W = = qTp-pTq 



□ 



Corollary 13.3. One has on 5 \ 

j[ — ZuJ = {pTq — qTp) + uJ {qYp — pYq) + 'uP' 
p{Tq — oJYq) — q{Tp — uJYp) + cU^ . 



Proof. We have 

Zcj _ ZW 

= w - ^^^ 

so the desired result follows immediately from (|13.24|1 . ()13.25() . 



□ 



The next lemma expresses a useful orthogonality property which enters several times in the 
computations of section ITU 

Lemma 13.4. Let X ,y he smooth vector fields on S, then on the set S \ one has 

Xqyp - Xpyq = . 
In particular, letting X = Y or T, and y = Z orY , we find 



Yq Zp — 


Yp Zq = 





Tq Zp — 


Tp Zq = 





Tq Yp — 


Tp Yq = 






Proof. To prove the lemma we note that 

xw = p Xp + q Xq , yw = p yp + q yq , 

and proceed as follows 

Xqyp - Xpy-q = X{qW-^)y{pW-^) - X{j)W^^)y{qW-^) 

= ^{{Xq-qXW){yp-pyW) - {Xp-pXW){yq-qyW)] 

_ —p^Xqyp — pqXqyq — pqXpyp - q^Xqyp + pqXpyp + q'^Xpyq + p^Xpyq + pqXqyq 
Xqyp - Xpyq _ q^Xqyp - p^Xpyq - q^Xqyp + p^Xpyq _ 

□ 

In the following lemma we collect some geometric identities involving the H-vnean curvature 
of S which play an essential role in the sequel. 

Lemma 13.5. One has on the set S \ 

(13.26) tXiP - pq{X2P + Xiq) + fX2q = H , 

(13.27) q Zp - p Zq = n , 

(13.28) Zp = qn , Zq = -pH . 
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The following formula is dual to 1)13. 26|) . 1)13. 27|) . 

(13.29) pqXip + fX2P - p^Xiq - p q = X2P - X^q , 

We also have the following expressions for the derivatives of the H-mean curvature along Y and 
T 

(13.30) q Y{Zp) - p Y{Zq) = YH , 

(13.31) q T{Zp) - p T{Zq) = TH . 
Proof. In view of l|13.7j) one has that l|13.26|l is equivalent to 

fX2q+tXip-pq{X2P + Xiq) = Xip + X2q , 

which is in turn equivalent to 

tX2q + fXip + p q{X2P + Xiq) = , 

and this is nothing but ()13.20|) . We now use 1)13. 26|) to prove 1)13. 27() as follows 

q Zp - p Zq = q^ Xip - pq X2P - pq Xiq + X2q = H . 

The proof of 1)13. 28|) immediately follows from the equation pZp + qZq = 0, from (|13.27j) . 
and from Cramer's rule. Next, it is easy to recognize that ()13.29|) is equivalent to ()13.19() . The 
proof of ()13.30() follows from differentiating (|13.27() with respect to Y , upon using Leibniz rule 
and Lemma ll3.4l Similarly, we establish (|13.31() by differentiating 1)13. 27|) with respect to T and 
then using Lemma 113.41 

□ 

We now establish a result which says that one of the two horizontal principal curvatures is 
zero. We stress that this phenomenon, whose Riemannian counterpart is obviously not true, 
reflects the fact that //-minimal surfaces are ruled surfaces, see |GPj . [UHMY] . 

Proposition 13.6. One has on S\ 

|Vi ' 1^1 I + IV2 ^ 1^2 I = [Zp) + {Zq) = U . 
In particular, if S is H -minimal, we have 

ivf'-'r.f |2 = \V^'^u^\^ = . 
Proof. According to (113.161) . (113.271) and (113.281) . we have 

_ |vf"5i,f|2 - |vf"5i/^|2 = {qZp-pZqf - {Zpf - {Zqf 
= tiZpf + fiZqf -2pqZpZq - {Zpf - {Zqf 
= - {f{Zpf + t{Zqf + 2pqZpZq) = - {pZp + qZqf = , 
where the last equation follows from Lemma Il3. 11 

□ 



Lemma 13.7. One has on 5 \ II5 

Zp Xi + Zq X2 = n Z , Zq Xi - Zp X2 = - HY 
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Proof. One easily obtains from the equations 

Zp Xi + Zq X2 = {pZp + qZq) Y + {qZp - pZq) Z 
= {jlZp — pZq) Z = H Z , 

where in the second to the last equality we have used ()13.17() . and in the last one we have used 
(|13.27jl . The proof of the second identity is similar. 

□ 

The next commutator formulas will be useful in the sequel. 

Lemma 13.8. One has on S\Tis 

[Z,Y] = T + n Z + {qYp - pYq) Y . 

Proof. To compute the commutator between Z and Y we use the equations ()13.9() and 1)13. 1U() 
to find 

[z,y] = Z{Y) - Y{Z) 

= qX,{pX^ + qX2) - pX^ipX^ + qX^) - pX^{qX^ - pX^) - qX^iqXi - pX^) 
= X1X2- X2X1 + {qXip - pXiq) Xi + {qX2P - pX2q) X2 
= T + {Zp + p{X2p-Xiq))Xi + {Zq + q{X2p- X{q))X2 
= T + Zp Xi + Zq X2 + {X2P - Xiq) Y , 

where we have repeatedly used (|13.17j) along with the identity + = 1. We now appeal to 
Lemma ri3.7l and to 1)13. 29() to reach the desired conclusion. 

□ 



Lemma 13.9. On S \ T,s, one has 

[Z, T] = {qTp - pTq) Y . 

Proof. Using the trivial commutation relations [Xj,T] = 0, i = 1,2 we obtain 

T{Z) = Tq Xi - Ip X2 + Z{T) . 

From this identity, and from ()13.13() . we obtain 

[Z,T] = TpX2 - TqXi = Tp{qY -pZ) - Tq{qZ +pY) 
= {qTp — pTq)Y — {pTp + q Tq) Z = {qTp — pTq)Y , 

where in the last equality we have used Lemma Il3. II 

Corollary 13.10. One has on S \ 

[T-ujY,Z] = oJ !^{T-ujY)+n Z^ . 

In particular, if S is H -minimal, then 

[T -ujY,Z] = lJ {T-LoY) . 



□ 
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Proof. One has from Lemmas 113.81 and 113.91 

[T - uJY, Z] = [T, Z]-io [Y, Z]+ZujY 

- {qTp - pTq)Y + UJ {T+ {qYp - pYq))Y -AY 
= uj T + {lJ {qYp — pYq) — {qTp — pTq) — A) Y , 

where we have used the hypothesis that S be i7-minimal. Using Corollary 113.31 we reach the 
desired conclusion. 

□ 



Corollary 13.11. If S is H -minimal, one has 

ZA = uj{lJ^-3A) 
For an arbitrary surface we have instead on S \ 



ZA 



Proof. From the assumption of ff-minimality of S, (|13.28|) and from C or ollar v 1 1 3 . 3 1 we obtain 
ZA = pZ{Tq- LoYq) - qZ{Tp - TvYp) + 2loZu) 
= p[Z, T - iJY]q - q[Z, T - uJY]p - 2uJA 

-2 3^) . 



UJ 



p{Tq - uJYq) - q{Tp - UYp) + 2A 



UJiUJ 



The proof of the second part of the corollary is based on a longer computation which, in 
addition, exploits also Corollary [TXTUl (|13.17() . (|13.27|) and (|13.28() . We leave the details to the 
interested reader. 

□ 



Corollary 13.12. One has on 5 \ 

[Z, Y\p = Tp + ZpH + {qYp - Wl) Yp , 
[Z, Y]q = Tq + Zqn + {qYp " pYq) Yq , 
[Z, T]p = {qTp — pTq) Yp , [Z, T]q = {qTp — pTq) Yq . 



Lemma 13.13. On 5 \ one has 

Z{qYp - pYq) - {qYp - pYqf - {qTp-pTq) = YH + Ti^ . 
Proof. Thanks to Lemma 113.41 we have 

Z{qYp - pYq) = q Z{Yp) - p Z{Yq) 
= qY{Zp) - pY{Zq) + q [Z,Y]p - p [Z,Y]q 

= YT-C + q{Tp+Zpn + {qYp - pYq)Yp} - p{Tq + Zq H + {qYp - pYq)Yq} 
= YTC + {qTp — pTq) + {qZp — pZq) 7i + {qYp — pY'q) ^ 

= YH + + {qTp-pTq) + {qYp-pYqf , 

where we have used (|13.3()|) and Corollary 113.121 in the third equality, and (|13.27|) in the second 
to the last equality. 

□ 
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Lemma 13.14. One has on the set S \ 

Z{q Tp — p Tq) = T7i + {q Tp — p Tq) {q Yp — p Yq) . 
Proof. Using Lemma 113.41 we obtain 

Z{q Tp — p Tq) = q Z{Tp) — p Z(Tq) 
= q T{Zp) - p T{Zq) + q [Z, T]p - p [Z, T]q . 

Now using Lemma 17.41 again we obtain from 1)13.271) 

TH = q T{Zp) - p T{Zq) , 

which, substituted in the above equation gives, along with Corollarv ll3.12| the desired result. 

□ 



14. First and second variation of the i^-perimeter in the Heisenberg group 



A fundamental tool in Riemannian geometry are the first and second variation formulas for the 
area functional. Consider a oriented hypersurface S C M", with Gauss map u : S ^ and 
denote by = Gx{S) the hypersurface obtained by deforming S in the normal direction with 
the one-parameter family of local diffeomorphisms Gx{x) = x + XC{x)iy{x), where C £ C^{S), 
and A E M is small. One has the following theorem, see for instance (10.12), (10.13) in |Gij . or 
also |MM| . |Hi|, IHMI and [Enn] . 

Theorem 14.1. The first variation of the area of S is given by the formula 

(14.1) ^Hn-i{Gx{S)) = [ H C dHn-i , 

d\ A=0 Js 

where H = ki + ... + indicates the sum of the principal curvatures of S. The second 

variation is given by 

(14.2) ^Hr,.,iGxiS))\^^^ = ^{iVCP + e (h^ - fjl^^^l') } ^^"-1 ' 

where V denotes the Levi-Civita connection on S, and it can be shown that XliLi ^-^ ihe 

sum of the squares of the principal curvatures of S. 

In this section we consider an oriented surface in the Heisenberg group H^, with non-unit 
Riemannian normal AT, and horizontal Gauss map , and compute the first and second varia- 
tion for general deformations of S. We observe that, in view of applications to the fundamental 
question of stability of if-minimal surfaces, it is important to be able to treat general deforma- 
tions, versus deformations along a specific direction. More precisely, we consider deformations 
of S given hy S ^ = J\{S), where 

(14.3) Jx{S) = S + XX = S + x{aXi + bX2 + kT 

where a,b,k £ Gq{S \ S^), and A £ R is a small parameter. Throughout this section, and 
the following one, we will continue to use the notations of section 1131 We recall that is 
endowed with a left-invariant Riemannian metric with respect to which {Xi, X2,T} constitute 
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an orthonormal basis with inner product < •,• >. Since no other inner product will be used, 
there will not be any confusion, for instance, with the standard Euclidean inner product of M^. 



Definition 14.2. Let S C M.^ be an oriented surface, consider the family of vector fields 
X = aXi + 6X2 + kT , with a,b,k G Cq(5 \ and the family of surfaces iS'^. We define the 
first variation of the H -perimeter with respect to the deformation 1)14. 3|) as 

= 0, then we say that S is stationary if {S; X) = 0, for every X. 



Classical minimal surfaces are stationary points of the perimeter (the area functional for 
graphs). It is natural to ask what is the connection between the notion of ff- minimal surface 
and that of i7-perimeter. The answer to this question is contained in the following result. To 
simplify the formulas we introduce the following notation 

A A^ ^ def , , < X,N > 

(14.4) F = pa + qb + iok - 



Theorem 14.3. Let 5 C be an oriented surface, then 
(14.5) = j n F dan ■ 

Ln particular, S is stationary if and only if it is H -minimal. 



Versions of Theorem ll4.3l have also been obtained independently by other people. An approach 
based on motion by iJ-mean curvature can be found in |BCj . When X = ai>^ + kT , then a proof 
based on CR-geometry can be found in jCHMYj . and |RRT] . [RR2] . We mention that Hladky 
and Pauls have recently proved in |HPj (with a different approach which does not directly use 
the first variation) that, for a wide class of sub-Riemannian spaces, a non-characteristic 
hypersurface is a critical point of the ff-perimeter if and only if it is //-minimal. 



Definition 14.4. Given an oriented surface 5 C I 
H-perimeter with respect to the deformation ()14.3|) as 



we define the second variation of the 



A=0 



Our main result in this section is the following theorem. 
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Theorem 14.5. The second variation of the H -perimeter with respect to the deformation of S 
given by (|14.3j) is expressed by the formula 

(14.6) vj^j{S;X) = j {qZa-pZb){Tk-LjYk) 



+ (Ta-uJYa) 
+ {Tb-iJYb) 



2qZk — q{ap + bq) — p{aq — bp) 



2pZk + p{ap + bq) — q{aq 

+ 2 {aq — bp) {qZa — pZb)uj 

+ {Za + pulZk)'^ + (Zb + quJZk)^ 

+ {a' + b')Zj' 

+ 2 Lj{aZa + bZb) + 2 oJ^ {ap + bq)Zk 



[qZa — pZb + {aq 



dan 



In order to prove Theorems 114.31 and 114.51 we develop some prehminary material which con- 
stitutes the necessary geometric backbone. We begin by deriving from Theorem 110.11 two in- 
tegration by parts formulas which play a fundamental role in this section and in the following 
one. 



Lemma 14.6. Let ( eC^iS\ S^), then 

Z( dan 



C oj dan ■ 

IS Js 

Proof. We begin by noting that, thanks to (|13.14() . ()13.9|) and ()13.1U|) . we can rewrite the two 
identities in (|in.l4|) in the form 



(14.7) 



(14.^ 



/ {q Zu + q u uj} dan = j p uTi dan , 
Js Js 

/ {p Zu + p uuj} dan = — q u 7i dan ■ 
Js Js 



Choosing u = q(^ in (|14.7|) . and u = p(^ in (|14.8|) . and adding the resulting equations, we 
obtain 

/ {(j)Zp + qZq) C + if+f) ZC} dan + I {f+f)CiJ dan = . 
Js Js 



IS JS 
From the latter equation, and from (|13.5|) . (|13.17() . we immediately reach the conclusion. 



□ 



Remark 14.7. We have above derived Lemma \l4-()\ f'rom Theorem \1U.1\ svecialized to . The 
two results are in fact equivalent. To see this suppose that the identity in Lemma \14.b\ hold. 
Applying it twice, once with the choice C, = qu, and the other with C, = pu, with u G Cq (5 \ S), 
we obtain (|14.7() and ()14.8|1 if we use the identities Zp = qTC, Zq = —pJi, in ()13.28|) . 



Another crucial integration by parts formula which we will need is (|1().18|) in Theorem II O.HI 
For the reader's convenience we combine this formula and Lemma 114.61 into a single statement. 
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Lemma 14.8. Let f G C^iS), C G C^{S \ S^), then 

f ZC dan = - C Zf dan - / / C ^ dcrn ■ 
Js Js Js 

j f{TQ-ujYC)daH = - I C{Tf-ZoYf)daH + [ fCoJH dan ■ 
In particular, if S is H -minimal, we find 

[ fiTC-uJYOdan = - [ C {Tf -uJYf)daH ■ 
Js Js 

After these preliminaries we turn to the proofs of the main results in this section. We first 
recall the representation formula for the ff-perimeter of 5 given in (|8.3|1 of Definition 18.41 

(14,.) .„(5,./^^.../^V^.., 

where da represents the standard surface measure on S. Next, we establish a simple lemma 
which provides the general expression for the first and second variation of the //-perimeter. We 
consider, for small values of A G M, a deformation of S of the type = J\{S) = S + XX, where 
X e C^{S\^s;'^^)- We denote by the non-unit Riemannian normal on 5^. Letting 

X^ig) = Xi{Jx{g)) , X^{g) = X^iJxio)) , geS, 

we consider the functions 

(14.10) p^=<N^,X^>, q^=<N^,X^>, = ^ {p^f + {q^Y . 

We stress that we are assuming that X is compactly supported away from the characteristic 
set of 5, so that the angle function W for S never vanishes on the support of X , see ()6.4() . 



Lemma 14.9. The first variation of the H -perimeter along the deformation J\{S) 
is given by the formula 



S + \X 



(14.11) VfiS;X) = 

The second variation is given by 



dx 



A=0 



dan 



p 



X dp 



d\ 



i d\ 



(14.12) 



Vfi{S;X) 



a d^p 



dA2 



A=0 



dp^ 
dX 



+ 



dq^ 
dX 



A=0 



a dp>- I 
' "ST + 



,X dq^ 
' dX 



dan 



dan 



A=0 



dUH 



A=0 



dan 



Proof. Because of the assumptions on X, the integrals in the right-hand sides of (|14.11|) . (|14.12|) 
are performed on a compact set which does not intersect S5. By a partition of unity we can thus 
reduce the analysis to a set 5 nO, where O C is an open neighborhood of a point g^ & S\T,s. 
We can thus assume that there exist an open set C ^, and a parametrization 6 : Q. ^ 
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such that cSnO = 9{Q). We suppose that the orientation of S is given by AT = 0^ A0„. Recalling 
that da = \6u A 6v\du A dv, we can, after projecting S onto Q, rewrite (|14.9|1 as follows 



(14.13) aniSnO) 
According to p4.1,Sj) we have 



W da 



\/ p'^ + q'^ du A dv . 



aniS^nO) = [ duAdv 



Observing that 



dW^ 
~dX 



and that W 



A=o 



W, we find 



A dp^ , A dq^ 



^ du A dv 



A dp^ , A dq^ 



which gives ()14.11() . To obtain ()14.12() we proceed analogously, observing that 



~d>?' 



„A d'^p^ , , 
P -d>?- + 1 



dX^ 



^A dp^ j_ ^A dq^ 
d\ 



+ 



AdV (^)- + (^ 



d\ 



dq^ 



dX 



A=0 



dan 



□ 



We now turn to the proof of the main results. Given an open set C M^, we denote by 
: $7 — > a parametrization of an oriented surface 



(14.14) 



{0{u,v) = {x{u,v),y{u,v),t{u,v)) G \ {u,v) G . 



We assume throughout that the orientation of S is given by the non-unit normal N — Oy^ A 
Using (|13.2|) we see that 

(14.15) e{u,v) = xiu,v)Xi{0{u,v)) + y{u,v)X2{e{u,v)) + t{u,v)T . 

From this equation we find 

9u = XuXi + yuX2 + tyT + xXi^u + yX2,u , 
with a similar expression for 9^. Keeping in mind that 

2 ' 



(14.16) 
we obtain 
(14.17) 



Xl^u 
Xl^v 



Vu rp 

2 ' 
' 2 ' 



X2,u 
X2,v 



2 

Xy rp 

2 



9u = XuXi+yuX2 + {tu + ^^^^)T 
9, = x„Xi + y,X2 + {t, + ^Mi) T . 
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(14.18) 



The non-unit outer Riemannian normal to S is thus given by 



VuXl - XuX2 , 

y^Xi - x„X2 . 



(14.19) 



AT 



\ A 



+ 



yXt) xy^ 



Xi 

X2 



— (^yutv yv^u '^i'^uyv ^vyu)^ Xi 

-\- ^Xytn Xutv ~\~ '^i^uyv Xi,yu)^ X2 

~l~ i^uyv x^y-n) T. 

We denote hy u = N /\N\ the Riemannian Gauss map of S. Keeping in mind (|13.3p . we see 
from (|14.19|) that 

p — yu^v yvtu ^{^uVv Xvyu) ^ 

Q Xytu Xytv ~\~ 2i.'^UyV ^yy^) ) 

^UJ — X^yv XvVu ■ 

We note at this moment that, given the assumption 9 G C'^{Q), the functions p, q,uj, W are of 
class C^(0), and that moreover p,q, uJ are of class C^{S \ S5). In what follows, given a function 
C defined in a neighborhood of 5, we will by abuse of notation denote with ("(u, t>) = C o 0(m, v) = 
({x{u,v),y{u,v),t{u,v)). The chain rule gives 

(14.21) — Cx ~l~ yu Cy ~^ Ct J Cv — ~^ yv Cy ~^ Ct • 

Using , we obtain from 1)14. 21|) 



(14.20) 



(14.22) 



Cu = Xu XiC + Vu ^2C + {tu + TC , 

C. = X, XiC + X2C + (ty + y^^) TC . 



In the sequel, it will be convenient to also have the expression of Cu, Cv with respect to the 
orthonormal frame {Z,Y,T}, where Y and Z are like in (|13.9jl . p3.1Uj) . From (|14.22|) and 
H13.13|) . we have 



(14.23) 



Cu = {xuP + yuq)YC + {xuq-yuP) ZC + (t„ + ^^V^) 

Cv = {xyp + yyq)YC + {x^q-yyp)ZC + (t^ + ^i^^^^) TC . 



We now fix functions a,b,k £ Cq^{S \ S5), and consider the vector field 

(14.24) X = aXi+bX2+kT. 

For small values of A G M, we let 5^ be the surface obtained by deforming S through the map 
J\ = Id + XX , so that 

(14.25) Jx{g) = g + XiaXi + bX2 + kT) , geS . 
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The parametric representation of S is given by 



+ \X 



(14.26) 
so that 

(14.27) 9^1 = 9^ + XXu, 9i = 9, + XX,, 

and therefore the non-unit Riemannian normal to the surface = Jx{S) is given by 

(14.28) = 9^ A 9^ = N + X {9^ A - 9^ A Xu) + X^ X^ A X^ . 
From (|14.28() we obtain 



(14.29) 



dX 



A=0 



3u A Xr,} — 9y A Xu , 



d^N^ 



dA2 



A=0 



2 Xu A Xy 



Using H14.16|) we find 



(14.30) Xu = auX^ + buX2 + [ku + ^^li-^ 1 T 



(14.31) 



Xy = Xi + by X2 + [K^ 1 T . 



From (tmH) . (tTCTj) . (|Ti:!TT|l and (fTm|l one has 

(14.32) 
dN^ 

dX A=o 

tv + ^^!L_^^ hu - (^u + ^ ^^" ^ by + (y„A;„ - UyK) - ^{XuUv - XyUu) \Xi 

+ \ [tu + ^ ^^" ^ fli; - ^it; + a„ + (x^A:^ - XnA:^,) + ^{XuVv - XyVu) \X2 

+ (y 

) + (x„6t, 



Equations (I14.29I1 . (114.3011 . (114.311) also give 



(14.33) 



dA2 



A=0 



, , , 6x„-ay^,^ / bxu-ayu 
[bu ky H - Oi, A;„ H 



X, 



+ 



^ 6x» - gj/M ^ / bxy-ayy 



^2 



+ {auby - aybu)T \ . 



We now let 
(14.34) 

for which we clearly have 



' = Xi{9^) = Xi - A I T , 
X^ = X2{9^) = X2 + A f T , 



(14.35) 



dXl 
~dX 



T , 



dX^ a d'^X^ 



dX 



T 



dX^ 



, 



~d^ 



. 



Consider the quantities in p4.1()|l . From (|14.35)) we find 



(14.36) 



dp^ 
~dX 



A=0 



< 



dN^ 



dX 



^^^.X,> --<N,T> 
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'2^ A 



(14.37) 



dp 



dA2 



A=0 



< 



d^N^ 



Similarly, we find 
(14.38) 



d\ 



dA2 
dN^ 



, dN>^ 
,Xi> - b < —— 
A=o dX 



A=0 



< 



(14.39) 



d 



dX^ 



A=0 d\ 
d^N^ 



,X2> + - <N,T> 

A=0 2 



A=0 



< 



dX^ 



dN^ 

,X2> + a < — — 
A=o dX 



A=0' 



Lemma 14.10. Letp^ and relative to the surface = Jx{S), where J\ is defined by ()14.25() . 

then 

^^A 

W{-{Zb + buJ)-quJ Zk + p{Tk- UYk)} 

-- W {{Za + a ZJ) +poJ Zk + q{Tk-ujYk)] 
Proof. Using (|14.32j) . (|14.36|) and the third equation in 1)14. 20j) . we obtain 
,4.40, 1^ 



dp^ 




~dX 


A=0 


dq^ 




Ix 


A=0 



tv H ] bu - {tu-\ ) bv + [yukv - Vvku) - b to . 



/ V 2 

We now use the equations 1)14. 22|) to express the derivatives bu,b^,ku, k^ in terms of derivatives 
Zb, Yb, Tb with respect to the orthonormal frame {Z, Y, T}. Ordering terms one finds 

dp^ 



dX 



A=0 



+ 



yXfj X yi) \ , •, / y x n x yn 

tv H ^ I [xuP + yuq) - ( i« H 7, ) [x^p + yj,t 

tv H 7, \Xuq - yuP) - U« H 7, \Xvq - yvP) 



Yb 
Zb 



VuixvP + yvq) - yv{xuP + VuC, 



VuixvQ - VvP) - VvixuQ - VuP) 



Uuty Uvtu 2 (.XuVv x^yii) 

Simplifying in the latter equation, gives 
dp^ 



Yk 
Zk 

Tk — b u) . 



dX 



A=0 



+ 



Yb 
Zb 



yXvtu Xytv ~\~ '^{XuUv Xvyuj J Q yVutv Uvtu '^{.XuUv X^yu) j p 

- (xuUv - Xvyu)pYk - {xuyv - XvyuJqZk + pWTk - b oj , 

where we have used (|14.20() . At this point we notice that, in view of ()14.20|) again, the coefficient 
of Yb vanishes, and we obtain from the remaining terms the following espression 

dp^ 



dX 



A=0 



W {-Zb-boj + pTk - uj{pYk + qZk)} 



which gives the first equation in the thesis of the lemma. In a similar fashion, we obtain the 

A=o' 

□ 



desired expression of ^ 
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From Lemma Il4. lUI we immediately obtain the following crucial result. 



Lemma 14.11. In the situation of Lemma \l^.li\ we have 

W'^ {{Tk - LoYk) + {qZa - pZh) + {qa - pbp} 



P 



dX 



dX 



A=0 



With Lemma 114.111 in hands we are ready to give the proof of Theorem I14.l-{l 



Proof of Theorem \14-^ Substituting the equation in Lemma ri4.11l in (|14.11|) of Lemma ri4.91 

we obtain 



(14.41) 



Vf{S;X) = j {Tk - LJYk}daH 

+ { {qZa — pZb) + {qa — pb)oJ} dan ■ 
Js 



In order to extract the geometry from 1)14. 41|) we need to convert the two integrals in the 
right-hand side into ones which involve only the functions a, b and k, and not their covariant 
derivatives along the orthonormal frame {Z,Y,T}. This is where we use Lemma 114.81 for the 
first time. Applying (|14.7|) . (|14.8|) we find 



(14.42) 



/ {q Za + q u> a} dan = 
Js 

/ {p Zb + p oJ b} dan = 
Js 

Furthermore, Lemma 114.81 gives 

(14.43) j {Tk - iJYk}daH = 
Combining (|14.42|) . H14.43|) with (|14.41() we obtain 

(14.44) Vf(5;A') = j H {p a + q b +uj k] dan ■ 



p a Ti dan , 



qbTL dan ■ 



uj k TL dan 



Recalling the definition (|14.4|) of we reach the desired conclusion. 



□ 



With Lemma Il4. lUI and some elementary computations, we obtain the following result which 
is useful in the proof of Theorem 114.51 since it provides the integrand of the second addend in 
the right-hand side of (|14.12j) . 

Lemma 14.12. In the situation of Lemma \14-l(j\ we have 



(14.45) 



dX 



A=0 



+ 



dq^ 
dX 



A=0 



= {Za+ pujZkf + [Zb+ qujZkf 

+ {Tk - ujYkf + {a^ + b^) uf" 

+ 2 {Tk- LoYk) {qZa - pZb + uj{aq - bp)) 

+ 2 uj{aZa + bZb) + 2 uj'^{ap + bq)Zk . 
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We finally turn to the computation of the first addend in the right-hand side of (|14.12() . i.e. 



A=0 



From (|14.37|) and from H14.32|) . H14.33|l we obtain 



A=o 



A=0' 



,,r> 



w 



2ky{pbu-qau) + 2ku{qay - pK) 



+ 



{bxy - ayy){pbu - qau) + {hxu - ayu)iqay - pK) 



{aq 



Now we compute the three expressions in square brackets in the right-hand side of ()14.46() . 
Using ()14.23|) we obtain 



(14.47) 



ipbu-qau) = {xuP + yuq){pYb- qYa) + {xuq - yuP){pZb - qZa) 
+ 6. + ^^^^^VpT6-gra), 



(14.48) 



{qa^ -pby) 



XvP + yyq){pYb- qYa) - {x^q - y^p){pZb - qZa) 
yxy xy-D \ ._ 



tv + 



{pTb - qTa) 



These formulas, combined with (|14.2Up . give 
(14.49) {bxy - ayy){pbu - qau) + (bxu - ayu){qay - pb^) 

= W \ p{ap + bq) {Tb - uJYb) - q{ap + bq) {Ta - uJYa) 



+ {bp — aq){pZb — qZa)uJ f . 
Again from 1)14. 23|) and (|14.20|) . we obtain 



(14.50) 
and 



(14.51) 

Formulas (|14.50|) . H14.51() give 



p {Ta — ujYa) + qtoZa 



q {Tb - iJYb) - pujZb 



W , 



W . 



(14.52) 



{aq - bp) 



{Uvau Uu^^v) ~l~ {^ubv Xybu) 



{aq - bp) 



p{Ta- uJYa) -q{Tb- uJYb) 



+ {qZa — pZb)uj 



W . 
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Finally, a (long) computation, based on H14.23|) . (|14.20|1 . I|14.47|l . (|14.48|) . and the identity 
1, give 



(14.53) 



2kvipbu - qau) + 2ku{qav - pby) 
= 2 I - {pZb - qZa) {Tk - ujYk) 

+ Zk p (Tb - LoYb) -q{Ta- ZJYa) 



From (|14.46|) . H14.49|) . (|14.52|) and 1)14. 53|) we finally conclude 
(14.54) 



A=0 



2 (pZ6 - gZa) {Tk - uJYk) 



2 Z/c 



p {Tb - iJYb) -q{Ta- uJYa) 



+ p{ap + 6g) {Tb — toYb) — q{ap + bq) {Ta — lvYo) 
+ {bp — aq) {pZb — qZa)uj 



+ {aq - bp) 



p{Ta- ujYa) -q{Tb- uJYb) 



+ {qZa — pZb)uj . 
We have thus proved the following lemma. 



Lemma 14.13. In the situation of Lemma \l^.li\ we have 

^A d?P^ _i_ „A dV " 

(14.55) * 



A=0 



2 (pZ& - qZa) {Tk - uJYk) 



+ {Ta-iJYa) 



+ {Tb-uJYb) 
+ 2 {aq — bp) {qZa — pZb)U . 
We can finally give the proof of Theorem 114.51 



2qZk — q{ap + bq) — p{aq — bp) 



2pZk + p{ap + bq) — q{aq 



Proof of Theorem \14-5\ Combining (|14.12|) in Lemma 114.91 with Lemmas 114.111 114.121 and 
114.131 we obtain the desired conclusion. 

□ 



15. The stability of i7-minimal surfaces 



Unfortunately, in its present form Theorem 114.51 is not as useful as one would wish. A 
completely analogous situation occurs in the Riemannian case, where one still needs to carefully 
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use intrinsic integration by parts to extract the geometry, see |BG(jrj . The main objective of 
this section is to give a geometric meaning to the second variation formula of Theorem ll4.5l We 
stress that for the sake of simphcity, and because of its relevance in the applications to stability, 
we state it for stationary points of the ff-perimeter functional (i^-minimal surfaces), but a more 
general formula containing the //-mean curvature H, along with its covariant derivatives, can be 
obtained with some additional work if we use the full form of the geometric identities in section 
1131 We begin with the relevant definition. 

Definition 15.1. Given an oriented surface S C , with = 0, we say that S is stable 
if it is stationary (i.e., H -minimal) , and if 

Vn{S;X) > 0, for every G (7^(5,11^). 

If there exists X ^ such that V^j{S; X) < 0, then we say that S is unstable. 

Our main result concerning the stability is contained in the following theorem. 

Theorem 15.2. Let S CM.^ be H -minimal, then 

where F is as in (jl4.4|) . As a consequence, S is stable if and only if the following stability 
inequality of Hardy type holds on S 



[ (iJ^ -2A)F^daH < [ iV^'-^Fp dan ■ 
Js Js 



Corollary 15.3. Every vertical plane S = {{x,y,t) \ ax + Py = 7}, with + 0^ ^ 0, is 

stable. 

Proof. Consider the defining function (j){x, y, t) = ax + (3y — "f. One has u) = Tcj) = 0, and 
therefore uj = A = Q. Since every plane in is //-minimal, we can apply Theorem ll5.21 to find 
for every vector field X = aXi + 6X2 + fcT G C^(5, M^) 



Vfi{S;X) = j iV^'^FfduH > 0. 



This proves the stability of S. We note explicitly that in the present situation 

^ aa + (5b 

□ 

Another interesting consequence of Theorem 115.21 is the following stability inequality for in- 
trinsic graphs. We recall that a surface 5 C is called an intrinsic Xi-graph according to 
|FSS3j provided that there exist an open set C and a function G (7^(17) such that S 

can be described by (x, y, i) = (0, n, v) o ((>{u, v)e\ = (0, u, v) o (0(n, v),0, 0). This means that S 
admits the parametrization 

6{u,v) = (^(j){u,v),u,v — —(j){u,v)^ , {u,v)£il.. 

If instead S can be parametrized by 

0{u,v) = (^u,(l){u,v),v + ^(p{u,v)^ , {u,v)£n, 
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then we say that S is an intrinsic X2-graph. We only discuss the case of an intrinsic Xi-graph, 
leaving to the reader to provide the trivial changes necessary to treat the case of X2-graphs. 

Given a function F denote by Bfj,{F) = + (pFy the linear transport equation, so that 
B(i,{4>) = 4>u + 4'4'v indicates the nonlinear inviscid Burger operator acting on (p. Since from 
H14.2U|) we obtain 



(15.1) p = 1 , q = -B4<P) , w = , W = ^l + B^{(l}Y , 
we see that the Riemannian normal to an intrinsic Xi -graph is given by 

(15.2) N = X, -B^{<l))X2-(t>.T . 

As a consequence of the first equality in ^5.1)1 we deduce that an intrinsic Xi-graph always 
has empty characteristic locus. Furthermore, again from (|15.1jl . and from (|15.2|1 . we see that if 
O is bounded then the iJ-perimeter of S is expressed by the functional 



aniS) = V{4>) = I ^l + B^{4>Y dudv , 



so that 

duH = ^1 + B^{<j))'^ dudv , 

see also |ASVj . 

Corollary 15.4. Let S be a C'^ H -minimal, intrinsic Xi-graph, then S is stable if and only if 



f <tt±Em F^ dudv < 



y/l + B^{<py - Juy^T+B^' 

where F is as in (|14.4|) . 

Proof. We begin by observing that, thanks to 1)15. 1() we have 

,H _ 1 _ B^{4>) 



dudv 



(15.3) 



(j.H\± _ _ B^{<i)) y- _ 1 V 



Given a function / on 5, by abuse of notation we continue to indicate with the same letter 
the function /(u, v) = f{9{u, v)). A simple use of the chain rule as in 1)14. 21|) gives 

/„ = <A„ Xif + X2f - (PTf , f, = 0, Xif + Tf , 

where in the right-hand sides of the latter equations we have written Xif for Xif o 0, and 
similarly for X2/, T/. Using the latter two equations and the second equation in (|15.3|) . we 
obtain 

(15.4) Zf = - , ^^^^^ . 

We now use (|15.4|) to compute A = —ZuJ. From the latter two equations in (|15.1|) . one has 

/ ^ \ 

A = 



\.Ji + B4W) 



B4M ^ ^ B^{(t>)B^{B4,m 



l + BMf (1+^40)2)2 
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(15.5) B, ( -M^] = - n, 

see [(iiSj . and also |BSVj . Using (|15.5j) . after some simple computations, we obtain that the 
condition that S be iif-minimal is expressed by 

B^B^m = . 

Substituting this equation in the above formula for A we conclude that 

_ B^{(t>v) 

Again from H15.1|) we finally obtain 



ZJ^ - 2 A 



l + B, 



To reach the desired conclusion we are left with using the latter equation in the stability 
inequality in Theorem 115.21 in combination with the expression of dan and with ()15.4|) . 

□ 



In pnm\ it was conjectured that the only stable intrinsic graphs in are the vertical 
planes. Using also the results in |DGJN3j . in (BSVj the authors have provided a positive answer 
to this conjecture. We next turn to the proof of Theorem 115.21 



Proof of Theorem \15.^ Since we want to extract a more geometrically meaningful formula 
from the general expression in Theorem 114.51 we will now make several reductions. First, 
expanding the three squares, and regrouping terms using repeatedly + q'^ = 1, we find for the 
integrand in the right-hand side of (|14.6jl 



Xntegrand = 2 [qZa — pZh) {Tk — coYk) 



+ (Ta-uJYa) 



+ (Tb-ZJYb) 



2qZk — q{ap + bq) — p{aq 



2pZk + p{ap + bq) — q{aq — bp) 

+ (pZa + qZb)'^ + uj'^{Zk)'^ + 2{pZa + qZb)uJZk 
+ 2Zj'^{ap + bq)Zk + 2Lj{aZa + bZb) 
+ (a^p"^ + b'^q'^ + 2p g ab)uj'^ 
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We easily obtain from the latter equation 



(15.6) 



Vfj{S;X) = 2 
+ 2 



/ {qZa — pZb) 
Js 



(Tk — uYk) dan 



+ I <; {Ta-uJYa) 



q {Ta - ujYo) +p{Tb- uJYb) 
q{ap + bq) — p{aq 



Zk dan 



+ {Tb — ujYb) p{ap + bq) — q{aq — bp 

+ {pZa + qZb+UZkf 

+ 2Lo'^{ap + bq)Zk + 2uj{aZa + bZb) 

+ {ap + bqf uJ^ > dan ■ 



Our final objective is to remove all derivatives from the functions o, b and k from the right- 
hand side of 1)15. 6() . This is somewhat delicate and involves some effort. The final product will 
be achieved by a repeated use of the basic integration by parts Lemma ll4.8l and of the geometric 
identities in section [T^ We begin by observing that, thanks to ()13.28() . the i7-minimality of S 
implies that 



qZa — pZb = Z{qa — pb) 



Using this observation, Lemma 114.81 and Corollary 113. lUl we obtain 



(15.7) 



y |2 {qZa - pZb) {Tk - UYk) dan 

= - 2 {qa-pb)Z{T -ujY)k dan - 2 / W{qa -pb){T -ujY)k dan 
Js Js 

= - 2 {qa-pb){T -ujY)Zk dau + 2 / {qa - pb)[T - UY, Z]k dan 
Js Js 

— 21 Lj{qa —pb){Tk — uJYk) daH 

Js 

= 2 j Zk {T- ujY){qa - pb) dan 

— 2 ZJk {T — oJY){qa —pb) daH + — 2 / ZJk {T — uJY){qa —pb) dan 

Js Js 

q{Ta - ujYo) - p{Tb - ujYb) 



Zk da 



+ 2 aZk {Tq - UYq) da - 2 bZk {Tp - ujYp) da 
Js Js 
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+ (Ta-uJYa) 



+ (Tb-uJYb) 



q{ap + bq) - p{aq 



p{ap + bq) — q{aq — bp 

+ {pZa + qZb + UZk)'^ 

+ 2Lo'^{ap + bq)Zk + 2Lj{aZa + bZb) 

+ {ap + bqf \ dan ■ 



It should be clear to the reader that we have made some interesting progress, since we have 
eliminated terms containing products of derivatives of a, b and k. However, we are still far from 
our final goal. We next use the //-minimality of S, and (|13.28j) again, to see that 



{ZFf 



{pZa + qZb + uoZk) — Ak 



pZa + qZb + ojZk] + A^k^ - 2Ak{pZa + qZb) - ioAZ{k^) , 



where F is as in (|14.4() and A is the function defined in Corollarv 113.31 This identity, the fact 
that pZa + qZb = Z(ap + bq) (the //-minimality of S), Corollarv 1 1 3 . 1 1 1 and Lemma Il4. 81 give 

(15.9) J (^Za + qZb + ZJZk^ dan = J (ZF)^ dan 

+ 21 AkZ{ap + bq) dan - / (^^ - 2A)LJ^k^ dan 
Js Js 

= / (ZF)2 dan + {2A- uj'^){uj'^k^ + 2ujk{ap + bq)) dan 
Js Js 

- 2 j A{ap + bq)Zk dan ■ 
Substitution of into (fTTsl) gives 

(15.10) Vu{S;X) = [ (ZF)^ dan + [ {2A-uj'^)iLJ^k'^ + 2ZJk{ap + bq)) dan 

Js Js 



+ 2 j aZk 
+ 2 j bZk 



{Tq — u>Yq) + p up' — pA 



dan 



{Tp — ujYp) + q uP' — qA 



dan 



+ 



j |(Ta-cjya) 



q{ap + bq) — p{aq — bp) 



+ (Tb-iJYb) 



p{ap + bq) - q{aq 



dan 



+ / ujZ{a^ + b^) dan + (ap + bqf uP dan ■ 
Js Js 
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We now claim 

{Tq - uJYq) + p Zj"^ -pA = - {Tp - ZJYp) +quj'^ -qA = . 

We only check the first of the two equations, leaving it to the reader to provide the details of 
the second one. Corollarv 113.31 and the identity + = 1 give 

pA = {Tq — LoYq) — p q{Tp — uJYp) +puP' 

= {Tq — ujYq) — if {Tq — ZoYq) — p q{Tp — ujYp) +puf' 



{Tq - ujYq) + p uP' - q 



q{Tq — ijjYq) + p{Tp — ujYp) 



= {Tq - ujYq) + p uP' , 

which proves the first identity. Using the claim in (|15.1fl|) , with a further application of Lemma 



114. 8( we obtain 

(15.11) Vu{S;X) = [ {ZFf dan + [ {2A-oj'^){uJ^k^ + 2ujk{ap + bq)) dan 

Js Js 



+ 



J I (Ta - ZJYa) 



q{ap + hq) — p{aq 



+ {Tb-iJYb) 



>daH 



p{ap + bq) - q{aq - b] 

+ {A- ij'^){a^ + b"^) d(TH + {ap + bqf uP dan ■ 
Js Js 

Completing the square in the second integral in the right-hand side of 1)15. 11() . we find 

(15.12) Vu{S;X) = f {ZFf dau + [ {2A-uj'^)F^ dan 

Js Js 

+ y I (To - iJYa) -2apq + b{p'^ - q^) 
+ {Tb-uJYb) 2bpq + a{p'^--f) jda^ 

+ / {A-u]'^){a'^ + b^) duH - 2 j {A -uj^){ap + bqf dan ■ 
Js Js 

The proof of the theorem will be completed if we can establish the following crucial claim 



(15.13) j S^{Ta-iJYa) - 2ap q + b{p'^ - q'^'' 



{Tb - oJYb) 



dan 



2bp q + a{p'^ — q^) 

+ {A- uj'^){a^ + b^) dan -2 {A- uj'^){ap + bqf dan = . 
Js Js 

To prove ()15.13() we proceed as follows. First, Lemma 114.81 gives 

— 2 / apq{Ta — ujYa) dan = / cP{T — ujY)(j}q) dan ■ 
Js Js 

Therefore, the coefficient of a? in the left-hand side of (|15.13jl is given by 



{T -ujY){pq) + {A-u}'^)-2{A-uj'')t = {T - u}Y){p q) + {A - uj''){t - 1) 
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where we have used the identity p + q = 1. Similarly, we have 



2 / hpq{Tb-uYh)daH = - [ b^{T -u]Y){pq) dan , 
Js Js 



hence the coefficient of b in the left-hand side of (|15.1i-{)) is given by 



{T-uJY)ip q) + {A-uJ^){t -f) 



-{T -WY){pq) + {A-uj'^)-2{A-u5'^)t = 

Finally, we have 
j ^{f^ - t){Ta - TJYa) + a{p'^ - t){Tb - T:jYb)^d(jH 

We thus see that the coefficient of ab in the left-hand side of (|15.13j) is given by 



4(cj2 -A)pq+{T- uJY){f - p2 







From these considerations, the claim 1)15. 13|) follows. We have thus completed the proof. 



□ 
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